Skip to main content

​
Overview

As your product data and data model changes over time, so too may the shape of the data you wish to share with customers. As your schema changes, Prequel can help you evolve the data model schema in destination systems. When you update your data model in Prequel, schemas in destination systems will be updated automatically on the next transfer to that destination. Prequel’s schema evolution capabilities are limited to a set of “safe” and non-destructive operations that should handle most schema evolution use cases without resulting in undesirable consequences in the destination system.
Model configuration validation enforces safe schema evolutionOnly model updates that result in permitted schema evolution operations are permitted. Non-allowed model updates will be rejected.

​
Limitations

  • Changing data types is not allowed. Many data destinations do not allow changes to data types. To avoid broken destination states, Prequel prevents changing data types on existing columns.
  • Reusing table names within the same destination is not allowed. Two models assigned to the same destination must write to different tables. Models assigned to different destinations may share a table name.
  • Changing the name_in_destination of the primary key or last modified timestamp column is not allowed.
  • Column position is not guaranteed When a column is added to an existing model, it is appended to the end of the destination table schema via ALTER TABLE ADD COLUMN, regardless of where it appears in the model config. Downstream pipelines should not be dependent upon column positioning.

​
How it works

Schema evolutions happen automatically. When a transfer is enqueued, the data model configured at that time (submitted_at) is recorded and assigned with that transfer. When the schema is dequeued (usually <1 minute later), the necessary schema evolutions are calculated and performed (e.g., new columns are added) before the latest batch of data is loaded into the destination. The need for schema evolution is determined by comparing the schema from the previous successful evolution to the schema in enqueued transfer. Any permissible schema evolution operations will then be performed to modify the destination schema as needed.