> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prequel.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Learn how and why Datasets are replicated from Sources

In Prequel Import, a Dataset is a materialized cache of data representing a connected Source table or custom SQL query. Datasets are maintained in the [Datalake](/import/core-concepts/datalake).

Every Dataset is defined along two independent dimensions: its **type**, which controls how Prequel Import detects changes, and its **method**, which controls how Prequel Import reads data from the Source.

<CardGroup cols={2}>
  <Card title="Dataset Type" icon="1" href="#1-dataset-type">
    **Dimension** or **Fact**. Determines how Prequel Import detects changes.
  </Card>

  <Card title="Dataset Method" icon="2" href="#2-dataset-method">
    **Table**, **SQL**, or **Glob**. Determines how Prequel Import reads from the Source.
  </Card>
</CardGroup>

## 1. Dataset type

A Dataset's type determines which columns Prequel Import uses to detect changes and whether it can sync incrementally. It applies the same way to every method. See [Change Detection](/import/advanced/change-detection) for how each type is synced.

<Tabs>
  <Tab title="Dimension" icon="tags">
    **Dimension tables** contain rows that may be updated over time. These typically represent **entities** such as users, products, or accounts. Prequel Import tracks changes and syncs updates to existing records.

    **Configuration fields**

    See the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full payload.

    <ParamField body="primary_key_column" type="string" required>
      The column used to uniquely identify records.
    </ParamField>

    <ParamField body="last_modified_column" type="string">
      The column Prequel Import uses to find rows changed since the last sync. When set, every sync after the first reads only rows modified since the previous run. When unset, every source row is read on each sync. Either way records are merged by `primary_key_column`, so this setting controls whether the source read is filtered by time, not whether rows are replaced or removed.
    </ParamField>

    <ParamField body="is_deleted_column" type="string">
      A boolean column the Source uses to flag deleted rows. Prequel Import reads this flag and propagates those deletions to the Datalake. Rows that are physically removed from the Source are not detected, whether or not this column is set.
    </ParamField>
  </Tab>

  <Tab title="Fact" icon="receipt">
    **Fact tables** are an append-only stream of **immutable events or transactions**, such as events, page views, or logs. New records are added but existing records are never modified.

    **Configuration fields**

    See the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full payload.

    <ParamField body="primary_key_column" type="string" required>
      The column used to uniquely identify records.
    </ParamField>

    <ParamField body="created_at_column" type="string" required>
      The column recording when each record was created. Required for efficient append-only change detection.
    </ParamField>
  </Tab>
</Tabs>

## 2. Dataset method

A Dataset's method determines how Prequel Import reads data from the Source.

<Tabs>
  <Tab title="Table" icon="table">
    Reads a named table from the Source, addressed by namespace and table name.

    **Configuration fields**

    See the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full payload.

    <ParamField body="source_table_namespace" type="string" required>
      The namespace (for example schema or database) containing the source table.
    </ParamField>

    <ParamField body="source_table_name" type="string" required>
      The name of the table in the Source.
    </ParamField>
  </Tab>

  <Tab title="SQL" icon="code">
    Reads the result set of a custom `SELECT` query you provide, so you can join, filter, or reshape data before it is replicated.

    **Configuration fields**

    See the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full payload.

    <ParamField body="query" type="string" required>
      The read-only `SELECT` statement Prequel Import runs against the Source.
    </ParamField>

    Prequel Import reads your query as a subquery, so a few constraints apply:

    * The query must return a result set. Use a `SELECT` or another read-only statement that returns rows. Statements that do not return rows, such as `INSERT` or `UPDATE`, are not supported.
    * Every column named in your type configuration (`primary_key_column`, `last_modified_column`, `is_deleted_column`, `created_at_column`) must appear in the query's result set. If a configured column is missing, the extract fails.
    * Rows with a null primary key are skipped, so make sure your query returns a non-null primary key for every row you want to ingest.

    <Note>
      For an incremental Dimension Dataset, the `last_modified_column` value must be stable and monotonically increasing so change detection advances correctly. Computed or non-monotonic timestamps can cause rows to be missed or re-read.
    </Note>

    ```json title="Dataset config" icon="brackets-curly" expandable theme={null}
    {
      "dataset": {
        "source_id": "1f3c9e7a-0b2d-4a8e-9c11-7d6f2b5a4c33",
        "method": "sql",
        "type": "dimension",
        "frequency_minutes": 60,
        "sql": {
          "query": "SELECT p.id, p.name, p.updated_at, c.label AS category FROM products p JOIN categories c ON c.id = p.category_id WHERE p.active = true"
        },
        "dimension": {
          "primary_key_column": "id",
          "last_modified_column": "updated_at"
        }
      }
    }
    ```
  </Tab>

  <Tab title="Glob" icon="asterisk">
    Reads object files that match a pattern. Available for object storage Sources.

    **Configuration fields**

    See the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full payload.

    <ParamField body="pattern" type="string" required>
      The glob pattern matching object files in the Source.
    </ParamField>

    <ParamField body="format" type="string" required>
      The format of the object files. One of `csv`, `parquet`, `jsonl`, `json`.
    </ParamField>
  </Tab>
</Tabs>

## Creating a dataset

A Dataset is created with `POST /import/datasets`. Along with one type block and one method block from above, every Dataset specifies these common fields.

**Configuration fields**

See the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full payload.

<ParamField body="name" type="string">
  A name for the Dataset.
</ParamField>

<ParamField body="source_id" type="uuid" required>
  The ID of the [Source](/import/core-concepts/sources) this Dataset reads from.
</ParamField>

<ParamField body="method" type="string" required>
  How Prequel Import reads from the Source. One of `table`, `sql`, `glob`.
</ParamField>

<ParamField body="type" type="string" required>
  The type of Dataset. One of `dimension`, `fact`.
</ParamField>

<ParamField body="frequency_minutes" type="integer" required>
  How often (in minutes) Prequel Import checks this Dataset for changes. Must be at least 1.
</ParamField>

<ParamField body="is_enabled" type="boolean">
  Whether the Dataset is enabled. Defaults to `true` when omitted.
</ParamField>

## Updating a dataset

After a Dataset is created, only its `name`, `frequency_minutes`, and `is_enabled` can be changed, with [`PATCH /import/datasets/{id}`](/import/api-reference/import-datasets/update-import-dataset). The type, method, query, and column configuration are fixed when the Dataset is created. To change a query or column mapping, create a new Dataset.