Skip to main content
One of Prequel’s core promises is the guarantee that data in the source will be accurately replicated in the destination. Every row will be transferred, no row will be dropped, and the destination will get all changes to existing rows. We want every user to build that confidence for themselves instead of having to take our word for it. This is why we surface data integrity checks, a way to audit the integrity of the data present in the destination, and compare it to what’s in the source.

​
Results

Data integrity checks return a range of information about the underlying destination table. Some results are informational, while others denote data integrity issues.

​
Limitations

Data integrity checks work by comparing hashes of certain values between the source and the destination. This allows them to be relatively compute and bandwidth efficient. In order to work across systems and in a way that doesn’t put undue load on those systems at scale, they are subject to a handful of limitations.

​
Supported systems

Data integrity checks are supported when both the source and the destination are one of the following systems:
  • Athena
  • Aurora Postgres
  • BigQuery
  • ClickHouse
  • Databricks
  • Delta Lake
  • Google Cloud Storage (beta)
  • MotherDuck
  • MySQL
  • Postgres
  • Redshift
  • Redshift Serverless
  • S3 (beta)
  • S3 compatible (beta)
  • Snowflake
  • SQL Server
Beta support for object storage destinationsData integrity checks on S3, Google Cloud Storage, and S3 compatible destinations are currently in beta.For Google Cloud Storage, only destinations using HMAC key authentication support data integrity checks today.

​
Other limitations

  • Data integrity checks only ensure the integrity of primary keys and last_modified_at timestamps. This is sufficient to build assurance of integrity.
  • On large tables, integrity checks leverage random sampling to build probabilistic confidence in the integrity of the data. This prevents the checks from overloading either system, and allows them to run in a finite time.
    • Automated sampling will likely occur on tables receiving more than 1B rows per day in volume.
    • You can also leverage manual sampling and specify a set of primary keys to compare across systems (regardless of table size).
  • Data Integrity jobs will not run at the same time as transfers to the same destination and vice versa. This is to prevent overloading the resources of the source or destination. This is also to ensure that integrity measurements are not taken on a “moving target”. If a data integrity check is in progress, all transfers to that destination will remain in PENDING state until the integrity check is complete.