Models

What are data models?

In this documentation, a data model refers to the Prequel representation of the structured data to be transferred using Prequel. An individual data model usually represents a single table, though (depending on the source/destination) it may also refer to a JSON blob or other file format (e.g., Parquet, CSV).

Data model configuration

📘

API reference

For the complete list of fields, their editability, and environment-specific constraints, see the Models API (PATCH) in the OpenAPI spec: OpenAPI 2023-12-01 – Models.

Each Prequel data model is configured via a JSON file with a specific format. If no model_name is provided, the name of the file (not including the extension) will be the name used for the table in the destination database. For example, for the configuration below, the destination table will appear as logs.

{
  "model_name": "logs",
  "columns": [
    {
      "name_in_source": "id",
      "name_in_destination": "id",
      "data_type": "text",
      "is_primary_key": true
    },
    {
      "name_in_source": "log",
      "name_in_destination": "event_log",
      "description":"A descriptive text entry of the event that occurred.",
      "data_type": "text"
    },
    {
      "name_in_source": "updated_at",
      "name_in_destination": "updated_at",
      "data_type": "timestamp",
      "is_last_modified": true
    }
  ],
  "source_table": "source_schema.application_logs",
  "source_name": "Example Production Source",
  "organization_column": "organization_id"
}

📘

Syncing config files from GitHub

You can use the Prequel GitHub Action to automatically sync any config file changes from GitHub to Prequel.

Configuration file keys

Each configuration file has 7 top level config keys: columns and source_table are always required; organization_column is required in multi-tenant tables mode; model_name, description, source_name, and source_id are optional.

Config file keys

📘

Choosing between source_name and source_id

Prefer source_name for config files shared across staging and production environments. Source names are consistent across environments, whereas source IDs are environment-specific UUIDs that differ between deployments. Use source_id when you need to reference a source by its UUID, or specify both fields together to require the source to match on both name and ID.

Column object keys

⚠️

is_primary_key and is_last_modified must appear within the first 32 columns

Both the primary key column and the last modified column must be positioned within the first 32 entries of the columns array (positions 1–32). This is a Prequel model requirement. It does not constrain the column order in your source table. The limit exists to respect ordinal position constraints in destination systems that implement indexing or partitioning by specific columns.

🚧

Using Acceptable Types

Use the correct data_type for each column to maintain compatibility with all destinations. Refer to the Data Types documentation for the complete list of supported types.

Column ordering

The order of columns in the columns array determines the column order in the destination table, but only at initial table creation. For existing destinations, columns added via schema evolution are always appended to the end of the destination table schema, regardless of their position in the model config. See Schema evolution for more detail.

Column name and table name conflicts

A column's name_in_source and name_in_destination cannot match the source table name or the destination table name (case-insensitive). This applies to both the column name as written and the table name as written.

Reserved column names

Some reserved keywords cannot be used as column names due to reserved statuses in common destination systems. There are 334:

Reserved column name prefixes

In order to avoid conflicts in common destination systems, column names cannot start with certain reserved prefixes. There are 10 reserved prefixes:

Grouping data models into products

By default, every destination you add to Prequel will be sent all configured models. Products allow you to only send a subset of those models to destinations.