Datasets

In Prequel Import, a Dataset is a materialized cache of data representing a connected Source table or custom SQL query. Datasets are maintained in the Datalake. Every Dataset is defined along two independent dimensions: its type, which controls how Prequel Import detects changes, and its method, which controls how Prequel Import reads data from the Source.

Dataset Type

Dimension or Fact. Determines how Prequel Import detects changes.

Dataset Method

Table, SQL, or Glob. Determines how Prequel Import reads from the Source.

1. Dataset type

A Dataset’s type determines which columns Prequel Import uses to detect changes and whether it can sync incrementally. It applies the same way to every method. See Change Detection for how each type is synced.

Dimension
Fact

Dimension tables contain rows that may be updated over time. These typically represent entities such as users, products, or accounts. Prequel Import tracks changes and syncs updates to existing records.Configuration fieldsSee the API reference for the full payload.

string

required

The column used to uniquely identify records. Must be a non-null string on every row. See Dataset requirements.

string

The column Prequel Import uses to find rows changed since the last sync. When set, every sync after the first reads only rows modified since the previous run. When unset, every sync reads the full Source. Records are merged by primary_key_column either way. This setting also determines whether rows removed from the Source are detected. See Deletions.

string

A boolean column the Source uses to flag deleted rows. Prequel Import delivers flagged rows as deletions. See Deletions for when this column is required.

2. Dataset method

A Dataset’s method determines how Prequel Import reads data from the Source.

Table
SQL
Glob

Reads a named table from the Source, addressed by namespace and table name.Configuration fieldsSee the API reference for the full payload.

string

required

The namespace (for example schema or database) containing the source table.

string

required

The name of the table in the Source.

Reads the result set of a custom SELECT query you provide, so you can join, filter, or reshape data before it is replicated.Configuration fieldsSee the API reference for the full payload.

string

required

The read-only SELECT statement Prequel Import runs against the Source.

Prequel Import reads your query as a subquery, so a few constraints apply:

The query must return a result set. Use a SELECT or another read-only statement that returns rows. Statements that do not return rows, such as INSERT or UPDATE, are not supported.
Every column named in your type configuration (primary_key_column, last_modified_column, is_deleted_column, created_at_column) must appear in the query’s result set. If a configured column is missing, the extract fails.

For an incremental Dimension Dataset, the last_modified_column value must be stable and monotonically increasing so change detection advances correctly. Computed or non-monotonic timestamps can cause rows to be missed or re-read.

Dataset config

{
  "dataset": {
    "source_id": "1f3c9e7a-0b2d-4a8e-9c11-7d6f2b5a4c33",
    "method": "sql",
    "type": "dimension",
    "frequency_minutes": 60,
    "sql": {
      "query": "SELECT p.id, p.name, p.updated_at, c.label AS category FROM products p JOIN categories c ON c.id = p.category_id WHERE p.active = true"
    },
    "dimension": {
      "primary_key_column": "id",
      "last_modified_column": "updated_at"
    }
  }
}

Reads object files that match a pattern. Available for object storage Sources.Configuration fieldsSee the API reference for the full payload.

string

required

The glob pattern matching object files in the Source.

string

required

The format of the object files. One of csv, parquet, jsonl, json.

Virtual columns for objectsObjects dropped into object storage often carry no row-level timestamps, which would force every sync to re-read every object. To solve this, each extracted record includes two virtual columns derived from the object’s storage metadata:

meta__filename (string): the key or path of the object the record was read from
meta__file_last_modified (timestamp): the object’s last modified time, as recorded by the storage provider

Set last_modified_column to meta__file_last_modified to sync incrementally at the object level: each sync reads only objects added or modified since the previous run. Note the granularity: if any part of an object changes, every record in that object is re-read.These columns are computed during extraction and are never written to your Source. They can also be referenced in Stream mappings.

Dataset requirements

Every record Prequel Import reads from the Source must meet these requirements, whatever the Dataset’s type or method.

A primary key on every record. The column named by primary_key_column must be present and non-null on every row the Dataset returns.
A string primary key. The primary_key_column must be a string type in the Source, such as text or varchar. Numeric and UUID columns need to be cast to a string, which the SQL method or a view in the Source can do.
Records under 32MB. The total size of each record must remain under 32MB to be processed.

Creating a dataset

A Dataset is created with POST /import/datasets. Along with one type block and one method block from above, every Dataset specifies these common fields. Configuration fields See the API reference for the full payload.

string

A name for the Dataset.

uuid

required

The ID of the Source this Dataset reads from.

string

required

How Prequel Import reads from the Source. One of table, sql, glob.

string

required

The type of Dataset. One of dimension, fact.

integer

required

How often (in minutes) Prequel Import checks this Dataset for changes. Must be at least 1.

boolean

Whether the Dataset is enabled. Defaults to true when omitted.

Updating a dataset

After a Dataset is created, only its name, frequency_minutes, and is_enabled can be changed, with PATCH /import/datasets/{id}. The type, method, query, and column configuration are fixed when the Dataset is created. To change a query or column mapping, create a new Dataset.

Deletions

How deletions reach your Destination depends on the Dataset type.

Dimension. When last_modified_column is not set, every sync reads the full Source, and rows that no longer appear are delivered as deletions. When last_modified_column is set, syncs are incremental and rows removed from the Source are not detected. In that case, the Source must flag deleted rows in the column named by is_deleted_column.
Fact. Fact tables are append-only, so deletions are not propagated.

On the Batch API, each deleted record carries prequel__is_deleted: true along with its primary key, or exposes {{.IsDeleted}} when you use a body template. On the Record API, use the {{.IsDeleted}} template variable.

Getting started

Core concepts

Features

Logging, Monitoring, and Observability

Sources

Security & compliance

Developer SDKs

Dataset Type

Dataset Method

1. Dataset type

2. Dataset method

Dataset requirements

Creating a dataset

Updating a dataset

Deletions

Dataset Type

Dataset Method

​1. Dataset type

​2. Dataset method

​Dataset requirements

​Creating a dataset

​Updating a dataset

​Deletions

1. Dataset type

2. Dataset method

Dataset requirements

Creating a dataset

Updating a dataset

Deletions