> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prequel.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Batch API

> Implementing a Prequel Destination for your batch-based receiving endpoint

<Note>
  A Batch-based API endpoint is the recommended implementation for most use cases. Contact the Prequel team if you have questions.
</Note>

A `webhook_batch` destination delivers multiple records in a single HTTP request.

For the API call to register a destination with Prequel, including a complete example, see [Create Destination](/import/import-api/overview).

## Authentication

Prequel signs every delivery request with the `X-Prequel-Webhook-Signature` header so your endpoint can validate the request originated from Prequel. Fetch the public key for verification from `GET /public/signatures/webhook-public-key`. See [Webhook headers](#webhook-headers) for the full header list and verification steps.

After verifying the request signature, you can optionally read `provider_id` and `load_id` from the envelope body to confirm the request belongs to an expected provider and load.

## Creating your destination spec

The `destination` object allows you to customize the contract between data shared by your customers and the shape Prequel provides to your receiving endpoint.

**Example:** Suppose you are importing a `users` table with string fields `id`, `email`, and `subscription`.

Below are four possible shapes for a `webhook_batch` destination. These are not exhaustive and features can be combined to confirm to your endpoint's requirements:

1. **Schema as body:** Records land in the `JSONL` artifact in the shape declared by `record_schema`, with no additional shaping.
2. **Custom body:** Use a `body` template to customize each record. Only supported when `format` is `json`.
3. **Routed by provider:** `uri` and/or `headers` vary at delivery time using the `ProviderID` delivery-level identifier.
4. **Destination per provider:** A separate destination is registered for each provider, with a `record_schema` that includes provider-specific custom fields.

<Tabs>
  <Tab title="Schema as body" icon="paper-plane">
    ```json title="POST destination payload" icon="brackets-curly" expandable theme={null}
    {
      "name": "users-batch-destination",
      "type": "webhook_batch",
      "record_schema": {
        "type": "object",
        "properties": {
          "id":    { "type": "string" },
          "email": { "type": "string" },
          "subscription": { "type": "string" }
        },
        "required": ["id", "email"]
      },
      "webhook_batch": {
        "request_template": {
          "method": "POST",
          "uri": "https://example.com/prequel/batch",
          "headers": { "Content-Type": "application/json" }
        },
        "format": "json"
      }
    }
    ```

    <Steps>
      <Step title="Name your destination">
        Set `name` (string, required) to a unique identifier for this destination within your account.
      </Step>

      <Step title="Set the destination type">
        Set `type` (string, required) to `webhook_batch` for batch deliveries.
      </Step>

      <Step title="Declare your record schema">
        Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery.
      </Step>

      <Step title="Configure the request template">
        Set `webhook_batch.request_template` (object, required) with the HTTP `method` (string, defaults to `POST`), `uri` (string, required), and optional `headers` (object). The `uri` and `headers` templates support the `.Prequel.*` namespace.
      </Step>

      <Step title="Pick a delivery format">
        Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces.
      </Step>

      <Step title="Tune optional throughput limits">
        Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Custom body" icon="pen-to-square">
    ```json title="POST destination payload" icon="brackets-curly" expandable theme={null}
    {
      "name": "users-batch-destination",
      "type": "webhook_batch",
      "record_schema": {
        "type": "object",
        "properties": {
          "id":    { "type": "string" },
          "email": { "type": "string" },
          "subscription": { "type": "string" }
        },
        "required": ["id", "email"]
      },
      "webhook_batch": {
        "request_template": {
          "method": "POST",
          "uri": "https://example.com/prequel/batch",
          "headers": { "Content-Type": "application/json" },
          "body": "{\"email\": \"{{.Record.email}}\", \"user\": {{.Record.AsJson}}, \"is_deleted\": {{.IsDeleted}}, \"provider_id\": \"{{.Prequel.ProviderID}}\", \"load_id\": \"{{.Prequel.LoadID}}\"}"
        },
        "format": "json"
      }
    }
    ```

    <Steps>
      <Step title="Name your destination">
        Set `name` (string, required) to a unique identifier for this destination within your account.
      </Step>

      <Step title="Set the destination type">
        Set `type` (string, required) to `webhook_batch` for batch deliveries.
      </Step>

      <Step title="Declare your record schema">
        Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery.
      </Step>

      <Step title="Configure the request template with a body template">
        Set `webhook_batch.request_template` (object, required) with `method`, `uri`, `headers`, and `body`. The `body` template renders once per record while Prequel encodes the `JSONL` artifact, with access to record fields, the deletion flag, and delivery-level identifiers. The `uri` and `headers` templates render once per delivery and have access to delivery-level identifiers only. See [Template variables](#template-variables) for the full list of available variables. Body templates are only valid when `format` is `json`; selecting `csv` or `parquet` rejects the destination.
      </Step>

      <Step title="Pick a delivery format">
        Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces.
      </Step>

      <Step title="Tune optional throughput limits">
        Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Routed by provider" icon="route">
    ```json title="POST destination payload" icon="brackets-curly" expandable theme={null}
    {
      "name": "users-batch-destination",
      "type": "webhook_batch",
      "record_schema": {
        "type": "object",
        "properties": {
          "id":    { "type": "string" },
          "email": { "type": "string" },
          "subscription": { "type": "string" }
        },
        "required": ["id", "email"]
      },
      "webhook_batch": {
        "request_template": {
          "method": "POST",
          "uri": "https://example.com/prequel/batch?provider={{.Prequel.ProviderID}}",
          "headers": { "Content-Type": "application/json" }
        },
        "format": "json"
      }
    }
    ```

    <Steps>
      <Step title="Name your destination">
        Set `name` (string, required) to a unique identifier for this destination within your account.
      </Step>

      <Step title="Set the destination type">
        Set `type` (string, required) to `webhook_batch` for batch deliveries.
      </Step>

      <Step title="Declare your record schema">
        Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery.
      </Step>

      <Step title="Route by provider in the request template">
        Set `webhook_batch.request_template` (object, required) with `method`, `uri`, and optional `headers`. Use `{{.Prequel.ProviderID}}` in `uri` and `headers` to route by the provider whose data is being delivered. These templates render once per delivery, before any records are selected. See [Template variables](#template-variables) for the full list of identifiers available.
      </Step>

      <Step title="Pick a delivery format">
        Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces.
      </Step>

      <Step title="Tune optional throughput limits">
        Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Destination per provider" icon="id-card">
    ```json title="POST destination payload" icon="brackets-curly" expandable theme={null}
    {
      "name": "acme-corp-batch-destination",
      "type": "webhook_batch",
      "record_schema": {
        "type": "object",
        "properties": {
          "id":      { "type": "string" },
          "email":   { "type": "string" },
          "subscription": { "type": "string" },
          "tax_id":  { "type": "string" }
        },
        "required": ["id", "email", "tax_id"]
      },
      "webhook_batch": {
        "request_template": {
          "method": "POST",
          "uri": "https://example.com/prequel/batch/acme-corp",
          "headers": { "Content-Type": "application/json" }
        },
        "format": "json"
      }
    }
    ```

    <Steps>
      <Step title="Register one destination per provider">
        Create a separate destination for each provider whose requirements differ. Give each one a `name` (string, required) that identifies the provider it serves.
      </Step>

      <Step title="Set the destination type">
        Set `type` (string, required) to `webhook_batch` for batch deliveries.
      </Step>

      <Step title="Declare a provider-specific record schema">
        Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields this provider sends, including any custom fields unique to them. Prequel validates records against this schema before delivery.
      </Step>

      <Step title="Configure the request template">
        Set `webhook_batch.request_template` (object, required) with `method`, `uri`, and optional `headers`. Because the destination is scoped to a single provider, you can hard-code provider-specific routing into `uri` and `headers` without needing template variables.
      </Step>

      <Step title="Pick a delivery format">
        Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces.
      </Step>

      <Step title="Tune optional throughput limits">
        Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Template variables

The `body` template has access to every variable below; the `uri` and `headers` templates only have access to the `.Prequel.*` namespace.

| Variable                     | Type      | Description                                                                                                          |
| ---------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------- |
| `{{.Record.<field>}}`        | `any`     | Value of a specific field from the record (e.g. `{{.Record.email}}`). The field must be declared in `record_schema`. |
| `{{.Record.AsJson}}`         | `object`  | The full record serialized as a JSON object.                                                                         |
| `{{.IsDeleted}}`             | `boolean` | `true` when Prequel is delivering a record deletion, `false` otherwise.                                              |
| `{{.Prequel.ProviderID}}`    | `string`  | Identifier for the provider whose data is being delivered.                                                           |
| `{{.Prequel.LoadID}}`        | `string`  | Identifier for the active load job.                                                                                  |
| `{{.Prequel.DestinationID}}` | `string`  | Identifier for the destination receiving this delivery.                                                              |
| `{{.Prequel.StreamID}}`      | `string`  | Identifier for the stream this delivery came from.                                                                   |

## Delivery format

Prequel makes one HTTP request per batch. The request body is an envelope with the following fields:

| Field          | Type      | Description                                                                                  |
| -------------- | --------- | -------------------------------------------------------------------------------------------- |
| `url`          | `string`  | Presigned URL to the batch file. The URL itself carries a 24-hour TTL set by object storage. |
| `format`       | `string`  | Format of the file at `url`. One of `json`, `csv`, or `parquet`.                             |
| `record_count` | `integer` | Number of records contained in the batch file.                                               |
| `batch_id`     | `string`  | Unique identifier for this batch.                                                            |
| `provider_id`  | `string`  | Identifier for the provider whose data is being delivered. Omitted when unavailable.         |
| `load_id`      | `string`  | Identifier for the active load job. Omitted when unavailable.                                |

Your endpoint reads the envelope, downloads the file at `url`, and processes the records inside. The `format` field on the destination controls what kind of file the URL points to:

| Format    | File at `url`                                                               |
| --------- | --------------------------------------------------------------------------- |
| `json`    | `JSONL`. Each line is one record, optionally shaped by the `body` template. |
| `csv`     | `CSV`. Body templates are not supported.                                    |
| `parquet` | `Parquet`. Body templates are not supported.                                |

## Webhook headers

Prequel includes the following headers on every delivery request.

| Header                        | Description                                                                                                                                                                                                                                               |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `X-Prequel-Webhook-Signature` | SHA-256 RSA PKCS1 v1.5 signature of the payload. To verify, fetch the public key from `GET /public/signatures/webhook-public-key` and check the hex-decoded signature against the SHA-256 of `<X-Prequel-Webhook-Timestamp>.<raw body>` using PKCS1 v1.5. |
| `X-Prequel-Webhook-Timestamp` | RFC 3339 timestamp of when the request was sent. Reject stale timestamps to prevent replay attacks.                                                                                                                                                       |
| `X-Prequel-Webhook-Digest`    | SHA-256 hash of the raw request body.                                                                                                                                                                                                                     |
| `Content-Type`                | Defaults to `application/json` unless overridden by the `headers` template.                                                                                                                                                                               |

## Response codes

Your endpoint should respond with an appropriate HTTP status code. Prequel uses the response to determine whether to retry delivery.

| Code      | Description                                                                                 |
| --------- | ------------------------------------------------------------------------------------------- |
| `2xx`     | Success. Prequel treats a 2xx response as a successful delivery.                            |
| `400`     | Bad Request. Malformed or missing fields. Prequel will not retry.                           |
| `403`     | Forbidden. Invalid signature or credentials. Prequel will not retry.                        |
| `404`     | Not Found. Endpoint or resource could not be found. Prequel will not retry.                 |
| `413`     | Payload Too Large. Request body exceeds your endpoint's size limit. Prequel will not retry. |
| `422`     | Unprocessable Entity. Validation failed for specific fields. Prequel will not retry.        |
| `429`     | Too Many Requests. Rate limit exceeded. Prequel will back off and retry.                    |
| `431`     | Request Header Fields Too Large. Prequel will not retry.                                    |
| Other 4xx | Other client error. Prequel will not retry.                                                 |
| `500`     | Internal Server Error. Unexpected error during processing. Prequel will retry.              |
| `502`     | Bad Gateway. Upstream error. Prequel will retry.                                            |
| `503`     | Service Unavailable. Prequel will retry.                                                    |
| `504`     | Gateway Timeout. Request timed out. Prequel will retry.                                     |
| Other 5xx | Other server error. Prequel will retry.                                                     |