Understanding Prequel data types
Because data can be imported from many different source systems, and because data types are not necessarily synonymous across data stores, Prequel Import normalizes every source column into a common Prequel type. The Prequel type determines how each value is represented once it lands in your datalake and how it is delivered to your destination.Source type mapping
Reference this table to see how each source column type is normalized into a Prequel type.Timestamps without a time zone are handled as RFC 3339 stringsSource timestamp types that do not carry a time zone are mapped to the Prequel
string type rather than timestamp. Assuming a time zone such as UTC would be incorrect and could mislead downstream consumers.This applies to:- BigQuery
datetime - Databricks
timestamp_ntz - Postgres
timestampandtimestamp without time zone - Redshift
timestampandtimestamp without time zone - Snowflake
datetime,timestamp, andtimestamp_ntz - Object storage and SFTP
timestamp
"2023-05-01T14:30:00").To convert these values into actual timestamps, use an expression in the Stream’s table mapping to parse the string and apply the appropriate time zone.Arrays must contain numbersThe
vector Prequel type is used for arrays of numbers, such as ML embeddings. Values are converted to 64-bit floating point numbers. Array types not listed in the vector row above are not supported.Arrays of strings, structs, or mixed types cause the import to fail. In a database Source, convert them with a SQL query or a view that serializes the column to a JSON string.Datalake representation
Records are stored in your datalake, and delivered to your destination, as JSON. Each Prequel type has a stable JSON representation.Decimals are represented as strings to preserve precisionValues of the
decimal type are serialized as numeric JSON strings rather than JSON numbers.This prevents the loss of precision that would occur if high precision decimals were represented as floating point numbers.