Writing config files

Configuration files overview

The Prequel data model is configured via a JSON file with a specific format. If no model_name is provided, the name of the file (not including the extension) will be the name used for the table in the destination database. For example, for the configuration below, the destination table will appear as logs.

{
  "model_name": "logs",
  "columns": [
    {
      "name_in_source": "id",
      "name_in_destination": "id",
      "data_type": "text",
      "is_primary_key": true
    },
    {
      "name_in_source": "log",
      "name_in_destination": "event_log",
      "description":"A descriptive text entry of the event that occured.",
      "data_type": "text"
    },
    {
      "name_in_source": "updated_at",
      "name_in_destination": "updated_at",
      "data_type": "timestamp",
      "is_last_modified": true
    }
  ],
  "source_table": "source_schema.application_logs",
  "source_name": "Example Production Source",
  "organization_column": "organization_id"
}

Configuration file keys

Each configuration file has 6 top level config keys: (3 required) columns, source_table, and organization_column, and (3 optional) model_name, description, and source_name.

Config file keys

Config Key	Required	Description
`model_name`		Name of table to be loaded into the destination. If left blank, the file name (not including file extension) will be used.
`description`		Table description. Will be applied to destination tables if provided.
`columns`	✅	A list of all destination columns (for this destination table). This is used to `CREATE` the table in the destination database, so it is important to use accurate types that correspond to the source data. The order of this list dictates the order in which columns appear in the destination database. These columns are also used to create the `READ` and `WRITE` query, so it is important to use the correct `name_in_source` and a descriptive `name_in_destination` for end customers. They can, but do not need to be, the same name. `is_primary_key` and `is_last_modified` are two important column flags that indicate which columns to use for detecting changes and propagating changes to the correct row in the destination. One of each must be present on one of the column objects.
`source_table`	✅	Specifies the name of the source table to be queried.
`source_name`		Specifies the source from which this model should read data. This field is only required if multiple sources exist in the account. If left blank and only one source exists, models will read from that source by default.
`organization_column`	✅	Specifies the column that will be used for filtering data by organization. Because this column is not necessarily a column that you want to transfer to users, it is not required within the `columns` list (unlike `is_primary_key` and `is_last_modified`). This column must be a string or varchar (non-numeric) data type.

Column object keys

Column object key	Required	Description
`name_in_source`	✅
`name_in_destination`	✅
`data_type`	✅
`is_primary_key`	(On exactly one column)
`is_last_modified`	(On exactly one column)
`description`

🚧
Using Acceptable Types
It is important to use the correct data_type for each column to ensure compatibility with all destinations. Refer to the Data Types documentation for the complete list of supported types.

By default, every destination you add to Prequel will be sent all the models defined in the prequel/models directory. Products allow you to only send a subset of those models to destinations.