Skip to main content

Understanding Prequel data types

Because data can be imported from many different source systems, and because data types are not necessarily synonymous across data stores, Prequel Import normalizes every source column into a common Prequel type. The Prequel type determines how each value is represented once it lands in your datalake and how it is delivered to your destination.

Source type mapping

Reference this table to see how each source column type is normalized into a Prequel type.
Timestamps without a time zone are handled as RFC 3339 stringsSource timestamp types that do not carry a time zone are mapped to the Prequel string type rather than timestamp. Assuming a time zone such as UTC would be incorrect and could mislead downstream consumers.This applies to:
  • BigQuery datetime
  • Databricks timestamp_ntz
  • Postgres timestamp and timestamp without time zone
  • Redshift timestamp and timestamp without time zone
  • Snowflake datetime, timestamp, and timestamp_ntz
  • Object storage and SFTP timestamp
These values are preserved as RFC 3339 formatted timestamp strings (e.g., "2023-05-01T14:30:00").To convert these values into actual timestamps, use an expression in the Stream’s table mapping to parse the string and apply the appropriate time zone.
Arrays must contain numbersThe vector Prequel type is used for arrays of numbers, such as ML embeddings. Values are converted to 64-bit floating point numbers. Array types not listed in the vector row above are not supported.Arrays of strings, structs, or mixed types cause the import to fail. In a database Source, convert them with a SQL query or a view that serializes the column to a JSON string.

Datalake representation

Records are stored in your datalake, and delivered to your destination, as JSON. Each Prequel type has a stable JSON representation.
Decimals are represented as strings to preserve precisionValues of the decimal type are serialized as numeric JSON strings rather than JSON numbers.This prevents the loss of precision that would occur if high precision decimals were represented as floating point numbers.
Precision limitations for floating point and integer typesFloating point values are stored in IEEE 754 binary format, which cannot represent all decimal fractions exactly. To ensure accuracy, we strongly recommend using fixed precision types such as decimal instead.Integer values must fit within a signed 64-bit range. Values above 9223372036854775807 will overflow.