DocumentationAPI Reference

Configuration

Configuring your data model.

Suggest Edits

Configuring your data model

Once one or more sources has been connected, the data model must be configured so that Prequel can send the right data in the right format to each destination. Configuration covers a few important considerations:

  • selecting data: selecting the right data for a given destination
  • transforming data: specifying the correct format of the data
  • typing data: specifying the destination type of the data
  • detecting changes: after the first sync, only transferring updated data

📘

Configure your data model

  • Create one model file per table to be transferred, and store them in your connected repository. To link your GitHub repository, refer to the documentation.

Configuration files overview

The Prequel data model is configured via a JSON file with a specific format. The name of the file (not including the extension) will be the name used for the table in the destination database. For example, for the configuration below, the destination table will appear as logs.

{
  "columns": [
    {
      "name_in_source": "id",
      "name_in_destination": "id",
      "data_type": "text",
      "is_primary_key": true
    },
    {
      "name_in_source": "log",
      "name_in_destination": "event_log",
      "data_type": "text"
    },
    {
      "name_in_source": "updated_at",
      "name_in_destination": "updated_at",
      "data_type": "timestamptz",
      "is_last_modified": true
    }
  ],
  "source_table": "source_schema.application_logs",
  "source_name": "Example Production Source",
  "organization_column": "organization_id"
}

Configuration file keys

Each configuration file has 4 top level config keys: columns, source_table, source_name, and organization_column.

Config keys

Config KeyDescription
columnsA list of all destination columns (for this destination table). This is used to CREATE the table in the destination database, so it is important to use accurate types that correspond to the source data. The order of this list dictates the order in which columns appear in the destination database.

These columns are also used to create the READ and WRITE query, so it is important to use the correct name_in_source and a descriptive name_in_destination for end customers. They can, but do not need to be, the same name.

is_primary_key and is_last_modified are two important column flags that indicate which columns to use for detecting changes and propagating changes to the correct row in the destination. One of each must be present on one of the column objects.
source_tableSpecifies the name of the source table. This is used to parse for the most recent update when detecting changes over time.
source_name[Optional]Specifies the source from which this model should read data. This field is only required if multiple sources exist in the account. If left blank and only one source exists, models will read from that source by default.
organization_columnSpecifies the column that will be used for filtering data by organization. Because this column is not necessarily a column that you want to transfer to users, it is not required within the columns list (unlike is_primary_key and is_last_modified). This column must be a string or varchar (non-numeric) data type.

🚧

Using Acceptable Types

It is important to use the correct data_type for each column to ensure compatibility with all destinations. Refer to the Data Types documentation for the complete list of supported types.

Updated 5 months ago