Writing config files

Configuration files overview

The Prequel data model is configured via a JSON file with a specific format. If no model_name is provided, the name of the file (not including the extension) will be the name used for the table in the destination database. For example, for the configuration below, the destination table will appear as logs.

{
  "model_name": "logs",
  "columns": [
    {
      "name_in_source": "id",
      "name_in_destination": "id",
      "data_type": "text",
      "is_primary_key": true
    },
    {
      "name_in_source": "log",
      "name_in_destination": "event_log",
      "description":"A descriptive text entry of the event that occured.",
      "data_type": "text"
    },
    {
      "name_in_source": "updated_at",
      "name_in_destination": "updated_at",
      "data_type": "timestamp",
      "is_last_modified": true
    }
  ],
  "source_table": "source_schema.application_logs",
  "source_name": "Example Production Source",
  "organization_column": "organization_id"
}

Configuration file keys

Each configuration file has 6 top level config keys: (3 required) columns, source_table, and organization_column, and (3 optional) model_name, description, and source_name.

Config file keys

Config Key	Required	Description
model_name		Name of table to be loaded into the destination. If left blank, the file name (not including file extension) will be used.
description		Table description. Will be applied to destination tables if provided.
columns		A list of all destination columns (for this destination table). This is used to CREATE the table in the destination database, so it is important to use accurate types that correspond to the source data. The order of this list dictates the order in which columns appear in the destination database. These columns are also used to create the READ and WRITE query, so it is important to use the correct name_in_source and a descriptive name_in_destination for end customers. They can, but do not need to be, the same name. is_primary_key and is_last_modified are two important column flags that indicate which columns to use for detecting changes and propagating changes to the correct row in the destination. One of each must be present on one of the column objects.
source_table		Specifies the name of the source table to be queried.
source_name		Specifies the source from which this model should read data. This field is only required if multiple sources exist in the account. If left blank and only one source exists, models will read from that source by default.
organization_column		Specifies the column that will be used for filtering data by organization. Because this column is not necessarily a column that you want to transfer to users, it is not required within the columns list (unlike is_primary_key and is_last_modified). This column must be a string or varchar (non-numeric) data type.

Column object keys

Column object key	Required	Description
name_in_source
name_in_destination
data_type
is_primary_key	(On exactly one column)
is_last_modified	(On exactly one column)
description

🚧

Using Acceptable Types

It is important to use the correct data_type for each column to ensure compatibility with all destinations. Refer to the Data Types documentation for the complete list of supported types.

By default, every destination you add to Prequel will be sent all the models defined in the prequel/models directory. Products allow you to only send a subset of those models to destinations.