# Set all destinations to 'enabled' state Source: https://docs.prequel.co/export/api-reference/actions/set-all-destinations-to-enabled-state /generated/openapi-generated-2023-12-01.json post /actions/enable-all Enable all destinations to be updated during future scheduled transfers. (Will not affect manual transfers initatied via the /transfer endpoint.) # Set all destinations to 'paused' state Source: https://docs.prequel.co/export/api-reference/actions/set-all-destinations-to-paused-state /generated/openapi-generated-2023-12-01.json post /actions/pause-all Pause all destinations so that they will not be updated during future scheduled transfers. (Will not affect manual transfers initatied via the /transfer endpoint.) # Generate Scoped Auth Token Source: https://docs.prequel.co/export/api-reference/auth/generate-scoped-auth-token /generated/openapi-generated-2023-12-01.json post /actions/generate-scoped-auth-token Generate a scoped auth token for a specific recipient. # Generate SSH Key Source: https://docs.prequel.co/export/api-reference/auth/generate-ssh-key /generated/openapi-generated-2023-12-01.json post /actions/generate-ssh-key Generate an SSH public key to authenticate into a bastion host when SSH tunneling is enabled. Unused keys are automatically deleted after 30 days. # Cancel specific data integrity job for a destination Source: https://docs.prequel.co/export/api-reference/data-integrity/cancel-specific-data-integrity-job-for-a-destination /generated/openapi-generated-2023-12-01.json post /destinations/{destination_id}/data-integrity/{job_id}/cancel Request cancellation of a data integrity job by ID. # Create data integrity job for a destination Source: https://docs.prequel.co/export/api-reference/data-integrity/create-data-integrity-job-for-a-destination /generated/openapi-generated-2023-12-01.json post /destinations/{destination_id}/data-integrity Enqueue a job to check data integrity. # Get specific data integrity job for a destination Source: https://docs.prequel.co/export/api-reference/data-integrity/get-specific-data-integrity-job-for-a-destination /generated/openapi-generated-2023-12-01.json get /destinations/{destination_id}/data-integrity/{job_id} Get a data integrity job by ID. # List data integrity jobs for a destination Source: https://docs.prequel.co/export/api-reference/data-integrity/list-data-integrity-jobs-for-a-destination /generated/openapi-generated-2023-12-01.json get /destinations/{destination_id}/data-integrity Get a list of data integrity jobs for a destination. # Create destination Source: https://docs.prequel.co/export/api-reference/destinations/create-destination /generated/openapi-generated-2023-12-01.json post /destinations Add a new destination. # Delete destination Source: https://docs.prequel.co/export/api-reference/destinations/delete-destination /generated/openapi-generated-2023-12-01.json delete /destinations/{destination_id} Delete an existing destination. # Get destination Source: https://docs.prequel.co/export/api-reference/destinations/get-destination /generated/openapi-generated-2023-12-01.json get /destinations/{destination_id} Get the specified destination. Destinations are the databases and data warehouses within the consumer environments. # Invalidate Delta Sharing tokens Source: https://docs.prequel.co/export/api-reference/destinations/invalidate-delta-sharing-tokens /generated/openapi-generated-2023-12-01.json post /destinations/{destination_id}/invalidate-delta-sharing Invalidate existing Delta Sharing JWTs for a destination. # Invalidate Delta Sharing tokens Source: https://docs.prequel.co/export/api-reference/destinations/invalidate-delta-sharing-tokens-1 /generated/openapi-generated-2023-12-01.json post /destinations/{destination_id}/sharing/delta/token/invalidate Invalidate existing Delta Sharing JWTs for a destination. # List destinations Source: https://docs.prequel.co/export/api-reference/destinations/list-destinations /generated/openapi-generated-2023-12-01.json get /destinations The Destination endpoint returns the configured data destinations. Destinations are the databases and data warehouses within the consumer environments. # List destinations for recipient Source: https://docs.prequel.co/export/api-reference/destinations/list-destinations-for-recipient /generated/openapi-generated-2023-12-01.json get /recipients/{recipient_id}/destinations Get a list of configured data destinations for a recipient. The recipient_id can be provided either via a recipient-scoped auth token or the `recipient_id` query parameter. # Test destination connection Source: https://docs.prequel.co/export/api-reference/destinations/test-destination-connection /generated/openapi-generated-2023-12-01.json post /destinations/{destination_id}/test-connection Test the connection to a destination. # Test new destination connection Source: https://docs.prequel.co/export/api-reference/destinations/test-new-destination-connection /generated/openapi-generated-2023-12-01.json post /test-destination Test the connection for a new destination before creating it. # Update destination Source: https://docs.prequel.co/export/api-reference/destinations/update-destination /generated/openapi-generated-2023-12-01.json patch /destinations/{destination_id} Update an existing destination. # API Status Source: https://docs.prequel.co/export/api-reference/health/api-status /generated/openapi-generated-2023-12-01.json get / Ping the Prequel API to make sure it is operating. # Create magic link Source: https://docs.prequel.co/export/api-reference/magic-links/create-magic-link /generated/openapi-generated-2023-12-01.json post /magic-links Add a new magic link. # Delete magic link Source: https://docs.prequel.co/export/api-reference/magic-links/delete-magic-link /generated/openapi-generated-2023-12-01.json delete /magic-links/{link_id} Delete an existing magic link. # Get magic link Source: https://docs.prequel.co/export/api-reference/magic-links/get-magic-link /generated/openapi-generated-2023-12-01.json get /magic-links/{link_id} Get the specified magic link. # List magic links Source: https://docs.prequel.co/export/api-reference/magic-links/list-magic-links /generated/openapi-generated-2023-12-01.json get /magic-links Retrieve list of magic links. # Create model Source: https://docs.prequel.co/export/api-reference/models/create-model /generated/openapi-generated-2023-12-01.json post /models Add a new model. # Delete model Source: https://docs.prequel.co/export/api-reference/models/delete-model /generated/openapi-generated-2023-12-01.json delete /models/{model_id} Delete an existing model. # Get model Source: https://docs.prequel.co/export/api-reference/models/get-model /generated/openapi-generated-2023-12-01.json get /models/{model_id} Retrieve an model. # List models Source: https://docs.prequel.co/export/api-reference/models/list-models /generated/openapi-generated-2023-12-01.json get /models Retrieve list of models. # Update model Source: https://docs.prequel.co/export/api-reference/models/update-model /generated/openapi-generated-2023-12-01.json patch /models/{model_id} Update an existing model. # Validate model Source: https://docs.prequel.co/export/api-reference/models/validate-model /generated/openapi-generated-2023-12-01.json post /models/{model_id}/validate Validate an model. # Get log Source: https://docs.prequel.co/export/api-reference/monitoring-&-logging/get-log /generated/openapi-generated-2023-12-01.json get /logs/{log_id} Returns the specified audit log. # List logs Source: https://docs.prequel.co/export/api-reference/monitoring-&-logging/list-logs /generated/openapi-generated-2023-12-01.json get /logs Returns the audit logs for all events. # API overview Source: https://docs.prequel.co/export/api-reference/overview Base URL, authentication, versioning, and response format for the Prequel Export API. The Prequel Export API is a REST API. All requests are made over HTTPS to your regional base URL and authenticate with an API key. ## Base URL | Deployment | Base URL | | :---------------- | :-------------------------- | | US (cloud-hosted) | `https://api.prequel.co` | | EU (cloud-hosted) | `https://eu-api.prequel.co` | | Self-hosted | Your configured host | ## Authentication Authenticate every request with your API key in the `X-API-KEY` header: ```bash title="Authenticated request" icon="terminal" theme={null} curl https://api.prequel.co/sources \ -H "X-API-KEY: $PREQUEL_API_KEY" ``` Prequel supports two credential types: * **API keys** grant full access to a single environment. A key is scoped to either staging (`sk_staging_...`) or production (`sk_prod_...`), and the key you use determines which environment the request reaches. See [Environments](/export/concepts/environments). * **Scoped auth tokens** are short-lived tokens (one-hour TTL) for client-side use with the [React SDK](/export/sdks/react/authentication). They grant limited, tenant-scoped access so a token can only read and write data for a single recipient. ## Versioning The current API version is `2023-12-01`. Pin your integration to it by sending the version header on every request. Include `X-Prequel-Api-Version: 2023-12-01` on all calls. See [API versioning](/export/api-reference/versioning) for the changes in this version and tips for migrating existing workflows. ## Request limits Requests may not exceed a **32 MB** payload. ## Response format Every response is a JSON object with three top-level fields: | Field | Description | | :-------- | :---------------------------------------------- | | `status` | The outcome of the request. | | `data` | The requested resource or resources. | | `message` | A human-readable message describing the result. | List endpoints are paginated. Paginated responses add `has_next` and `next_url`; follow `next_url` until `has_next` returns `false`. See [Pagination](/export/api-reference/versioning#pagination) for details. ## Next steps Generate a ready-to-use Postman collection from the OpenAPI spec. Review the changes in version 2023-12-01 and how to migrate. # Create a Postman collection Source: https://docs.prequel.co/export/api-reference/postman Generate a Postman collection from the Prequel Export OpenAPI spec. You can generate a ready-to-use Postman collection from Prequel's OpenAPI 3.1 spec, then use it to explore and test every Export API endpoint. ## Prerequisites * Postman v10 or later. * The Prequel Export OpenAPI spec URL: `https://docs.prequel.co/generated/openapi-generated-2023-12-01.json` In Postman, click the **Import** button in the left sidebar. 1. Paste the spec URL: `https://docs.prequel.co/generated/openapi-generated-2023-12-01.json` 2. Select **Postman Collection** as the import format. 3. Click **View Import Settings** and disable **Enable optional parameters** and **Include deprecated properties**. 4. Click **Import**. Postman generates a collection with one request per API operation. Add a new Postman environment with these variables, then reference them from each request: | Variable | Value | | :-------- | :------------------------------------------------------------------------------------------------- | | `apiKey` | Your staging (`sk_staging_...`) or production (`sk_prod_...`) API key. | | `baseUrl` | Your API endpoint: `https://api.prequel.co`, `https://eu-api.prequel.co`, or your configured host. | Reference `{{apiKey}}` from each request's `X-API-KEY` header and `{{baseUrl}}` as the request base URL. ## Next steps * Expand the collection and send a request to confirm your environment works. * Enhance the collection with examples, tests, or pre-request scripts. * Re-import the spec periodically to pick up API updates. # Create product Source: https://docs.prequel.co/export/api-reference/products/create-product /generated/openapi-generated-2023-12-01.json post /products Add a new product. # Delete product Source: https://docs.prequel.co/export/api-reference/products/delete-product /generated/openapi-generated-2023-12-01.json delete /products/{product_id} Delete an existing product. # Get product Source: https://docs.prequel.co/export/api-reference/products/get-product /generated/openapi-generated-2023-12-01.json get /products/{product_id} Retrieve an product. # List products Source: https://docs.prequel.co/export/api-reference/products/list-products /generated/openapi-generated-2023-12-01.json get /products Retrieve list of products. # Update product Source: https://docs.prequel.co/export/api-reference/products/update-product /generated/openapi-generated-2023-12-01.json patch /products/{product_id} Update an existing product. # Validate product Source: https://docs.prequel.co/export/api-reference/products/validate-product /generated/openapi-generated-2023-12-01.json post /products/{product_id}/validate Validate an product. # Create connection test of a recipient's source credentials Source: https://docs.prequel.co/export/api-reference/recipient-source-credentials/create-connection-test-of-a-recipients-source-credentials /generated/openapi-generated-2023-12-01.json post /recipient-source-credentials/{recipient_source_credential_id}/test-connection Test the connection of a recipient's source credentials. # Create source credential for a recipient Source: https://docs.prequel.co/export/api-reference/recipient-source-credentials/create-source-credential-for-a-recipient /generated/openapi-generated-2023-12-01.json post /recipient-source-credentials Add a new source credential for a recipient. # Delete a recipient's source credentials Source: https://docs.prequel.co/export/api-reference/recipient-source-credentials/delete-a-recipients-source-credentials /generated/openapi-generated-2023-12-01.json delete /recipient-source-credentials/{recipient_source_credential_id} Delete existing source credentials for a recipient. # Get a recipient source credential by ID Source: https://docs.prequel.co/export/api-reference/recipient-source-credentials/get-a-recipient-source-credential-by-id /generated/openapi-generated-2023-12-01.json get /recipient-source-credentials/{recipient_source_credential_id} The Recipient Source Credential endpoint returns the specified recipient source credential. Source credentials are the credentials used to connect to the source on behalf of a recipient. # List a recipient's source credentials Source: https://docs.prequel.co/export/api-reference/recipient-source-credentials/list-a-recipients-source-credentials /generated/openapi-generated-2023-12-01.json get /recipient-source-credentials The Recipient Source Credentials endpoint returns the configured source credentials for a recipient. Source credentials are the credentials used to connect to the source on behalf of a recipient. # Update a recipient's source credentials Source: https://docs.prequel.co/export/api-reference/recipient-source-credentials/update-a-recipients-source-credentials /generated/openapi-generated-2023-12-01.json patch /recipient-source-credentials/{recipient_source_credential_id} Update existing source credentials for a recipient. # Create recipient Source: https://docs.prequel.co/export/api-reference/recipients/create-recipient /generated/openapi-generated-2023-12-01.json post /recipients Add a new recipient. # Delete recipient Source: https://docs.prequel.co/export/api-reference/recipients/delete-recipient /generated/openapi-generated-2023-12-01.json delete /recipients/{recipient_id} Delete an existing recipient. # Get recipient Source: https://docs.prequel.co/export/api-reference/recipients/get-recipient /generated/openapi-generated-2023-12-01.json get /recipients/{recipient_id} Get the specified recipient. Recipients are the customers who will consume data. # List recipients Source: https://docs.prequel.co/export/api-reference/recipients/list-recipients /generated/openapi-generated-2023-12-01.json get /recipients The Recipient endpoint returns the configured data recipients. Recipients are the customers who will consume data. # Update recipient Source: https://docs.prequel.co/export/api-reference/recipients/update-recipient /generated/openapi-generated-2023-12-01.json patch /recipients/{recipient_id} Update an existing recipient. # List transfer reports Source: https://docs.prequel.co/export/api-reference/reports/list-transfer-reports /generated/openapi-generated-2023-12-01.json get /reports/transfers Retrieve transfer stats. Will return the summary stats of all destinations over the most recent 2 weeks by default. # Create sensitive trace endpoint Source: https://docs.prequel.co/export/api-reference/sensitive-trace-endpoints/create-sensitive-trace-endpoint /generated/openapi-generated-2023-12-01.json post /sensitive-trace-endpoints Add a new sensitive trace endpoint. # Delete sensitive trace endpoint Source: https://docs.prequel.co/export/api-reference/sensitive-trace-endpoints/delete-sensitive-trace-endpoint /generated/openapi-generated-2023-12-01.json delete /sensitive-trace-endpoints/{endpoint_id} Delete an existing sensitive trace endpoint. # Get sensitive trace endpoint Source: https://docs.prequel.co/export/api-reference/sensitive-trace-endpoints/get-sensitive-trace-endpoint /generated/openapi-generated-2023-12-01.json get /sensitive-trace-endpoints/{endpoint_id} Get the specified sensitive trace endpoint. # List sensitive trace endpoints Source: https://docs.prequel.co/export/api-reference/sensitive-trace-endpoints/list-sensitive-trace-endpoints /generated/openapi-generated-2023-12-01.json get /sensitive-trace-endpoints Retrieve list of sensitive trace endpoints. # Update sensitive trace endpoint Source: https://docs.prequel.co/export/api-reference/sensitive-trace-endpoints/update-sensitive-trace-endpoint /generated/openapi-generated-2023-12-01.json patch /sensitive-trace-endpoints/{endpoint_id} Update an existing sensitive trace endpoint. # Get object storage manifest signature public key Source: https://docs.prequel.co/export/api-reference/signatures/get-object-storage-manifest-signature-public-key /generated/openapi-generated-2023-12-01.json get /public/signatures/object-storage-manifest-public-key Get the public key you can use to verify object storage manifests. # Get webhook signature public key Source: https://docs.prequel.co/export/api-reference/signatures/get-webhook-signature-public-key /generated/openapi-generated-2023-12-01.json get /public/signatures/webhook-public-key Get the current public key used to verify webhook signatures. Clients should fetch this value instead of hardcoding it because the key may change. # Create source Source: https://docs.prequel.co/export/api-reference/sources/create-source /generated/openapi-generated-2023-12-01.json post /sources Add a new source. # Delete source Source: https://docs.prequel.co/export/api-reference/sources/delete-source /generated/openapi-generated-2023-12-01.json delete /sources/{source_id} Delete an existing source. # Get source Source: https://docs.prequel.co/export/api-reference/sources/get-source /generated/openapi-generated-2023-12-01.json get /sources/{source_id} The endpoint returns the details of a specific source by the given ID. # List sources Source: https://docs.prequel.co/export/api-reference/sources/list-sources /generated/openapi-generated-2023-12-01.json get /sources The sources endpoint returns the configured data sources. Sources are the databases or data warehouses within the provider environment. # Test source connection Source: https://docs.prequel.co/export/api-reference/sources/test-source-connection /generated/openapi-generated-2023-12-01.json post /sources/{source_id}/test-connection Test the connection to a source. # Update source Source: https://docs.prequel.co/export/api-reference/sources/update-source /generated/openapi-generated-2023-12-01.json patch /sources/{source_id} Update an existing source. # Cancel transfer Source: https://docs.prequel.co/export/api-reference/transfers/cancel-transfer /generated/openapi-generated-2023-12-01.json post /transfers/{transfer_id}/cancel Cancel a transfer that is still PENDING. Once a transfer is running, this will error. # Create transfer Source: https://docs.prequel.co/export/api-reference/transfers/create-transfer /generated/openapi-generated-2023-12-01.json post /destinations/{destination_id}/transfer Enqueue a transfer to specified destination. # Get transfer Source: https://docs.prequel.co/export/api-reference/transfers/get-transfer /generated/openapi-generated-2023-12-01.json get /transfers/{transfer_id} Retrieve details about a specific transfer. # List transfers Source: https://docs.prequel.co/export/api-reference/transfers/list-transfers /generated/openapi-generated-2023-12-01.json get /transfers Retrieve list of transfers. # List transfers for a given destination Source: https://docs.prequel.co/export/api-reference/transfers/list-transfers-for-a-given-destination /generated/openapi-generated-2023-12-01.json get /destinations/{destination_id}/transfers Retrieve list of transfers. # Generate Policy Source: https://docs.prequel.co/export/api-reference/trust-policy/generate-policy /generated/openapi-generated-2023-12-01.json post /actions/generate-aws-trust-policy Generate a trust policy for a recipient. # API versioning Source: https://docs.prequel.co/export/api-reference/versioning Key changes in API version 2023-12-01 and how to migrate existing workflows. The current Prequel Export API version is `2023-12-01`. Pin your integration to it by sending the version header on every request: ```bash title="Version header" icon="terminal" theme={null} X-Prequel-Api-Version: 2023-12-01 ``` Below are the key changes from the base API version to `2023-12-01` and tips for migrating existing workflows. ## Pagination List endpoints now return paginated responses instead of the full result set. The default page size is 10 items, with a maximum of 100. Paginated responses include `has_next` and `next_url`. To read a full list, follow `next_url` until `has_next` returns `false`. ## Vendor-shaped configuration Source and destination configuration moved from a flat object to a nested, vendor-specific object. For example, a Postgres source now nests its connection fields under a `postgres` key rather than at the root of the request body. ## Request format changes * **Recipients** no longer accept a `schema` field and instead use `id_in_provider_system` for tenant identification. * **Magic links** now require an existing `recipient_id` rather than creating a recipient inline. ## Endpoint simplification Routes with an `/export/` prefix are deprecated. Use the non-prefixed endpoints (for example, `/destinations` and `/sources`) instead of `/export/destinations` and `/export/sources`. ## Migration checklist Send `X-Prequel-Api-Version: 2023-12-01` on every request. Update any code that reads list endpoints to follow `next_url` until `has_next` is `false`. Move source and destination connection fields into their vendor-specific object. Replace `schema` with `id_in_provider_system` on recipients, and pass an existing `recipient_id` when creating magic links. Point requests at the non-prefixed endpoints. # Create webhook Source: https://docs.prequel.co/export/api-reference/webhooks/create-webhook /generated/openapi-generated-2023-12-01.json post /webhooks Add a new webhook destination to monitor transfer failures. All errors resulting from transfers that do not succeed will be delivered to this webhook endpoint for monitoring or logging. # Delete webhook Source: https://docs.prequel.co/export/api-reference/webhooks/delete-webhook /generated/openapi-generated-2023-12-01.json delete /webhooks/{webhook_id} Delete an existing webhook. # Get webhook Source: https://docs.prequel.co/export/api-reference/webhooks/get-webhook /generated/openapi-generated-2023-12-01.json get /webhooks/{webhook_id} The endpoint returns the details of a specific webhook by the given ID. # List webhooks Source: https://docs.prequel.co/export/api-reference/webhooks/list-webhooks /generated/openapi-generated-2023-12-01.json get /webhooks The webhooks endpoint returns the configured data webhooks. A webhook is an HTTP-based callback function. # Update webhook Source: https://docs.prequel.co/export/api-reference/webhooks/update-webhook /generated/openapi-generated-2023-12-01.json patch /webhooks/{webhook_id} Update an existing webhook. # Data types Source: https://docs.prequel.co/export/concepts/data-types Understanding Prequel data types ## Understanding Prequel data types Because data from any source can be synced to any destination, and because data types are not necessarily synonymous across data stores, a common mapping must be used to specify the source data type and predict the destination data type. As part of Prequel configuration, the expected data type must be defined ahead of time. ## Source type mapping If you are configuring a source and need to decide what Prequel type to use, reference this table. | Prequel Type | Supported | Athena | BigQuery | Clickhouse | Databricks | MongoDB | MotherDuck | MySQL | Postgres | Oracle | Redshift | Snowflake | SQL Server | | ------------ | --------------------- | --------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------- | --------------------------- | -------------------- | ------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------- | | `bytes` | | `binary` | `bytes` | `blob`, `mediumblob`, `longblob` | `binary` | `binData` | `binary`, `blob` | `binary`, `varbinary`, `blob`, `mediumblob`, `longblob` | `bytea` | `blob` | `VARBYTE`, `VARBINARY`, `BINARY VARYING` | `binary`, `varbinary` | `binary`, `image`, `varbinary` | | `string` | | `char`, `varchar`, `string` | `string` | `string`, `fixedstring`, `text`, `mediumtext`, `longtext`, `varchar` | `string` | `String`, `ObjectId` | `text`, `varchar` | `string`, `char`, `text`, `longtext`, `varchar` | `char`, `character`, `character varying`, `text`, `uuid`, `varchar` | `nchar`, `char`, `varchar2`, `nvarchar2` | `CHAR`, `CHARACTER`, `NCHAR`, `BPCHAR`, `CHARACTER VARYING`, `NVARCHAR`, `TEXT` | `CHAR`, `CHARACTER`, `NCHAR`, `BPCHAR`, `CHARACTER VARYING`, `NVARCHAR`, `TEXT` | `char`, `varchar`, `text`, `nvarchar`, `ntext`, `nchar` | | `boolean` | | `boolean` | `bool` | `bool`, `boolean` | `boolean` | `Boolean` | `boolean` | `boolean` | `boolean` | `number` | `boolean` | `boolean` | `bit` | | `integer` | | `smallint`, `integer` | N/A | `int`, `int8`, `int16`, `int32` | `smallint`, `int` | `Int32` | `smallint`, `int` | `smallint`, `mediumint`, `int` | `smallint`, `integer` | `int` | `smallint`, `integer` | `smallint`, `integer` | `smallint`, `int` | | `bigint` | | `bigint` | `int64`, `int`, `smallint`, `integer`, `bigint` | `int64`, `bigint` | `bigint` | `Int64`, `Long` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | | `decimal` | | `decimal` | `decimal`, `numeric` | `decimal`, `numeric` | `decimal`, `dec`, `numeric` | `Decimal128` | `decimal` | `decimal`, `numeric` | `decimal`, `numeric` | `decimal` | `decimal`, `numeric` | `decimal`, `numeric` | `decimal`, `numeric` | | `float` | | `real`, `float` | `float64` | `float32`, `float64`, `double` | `float`, `double` | `Double` | `float`, `double`, `float4`, `float8` | `float`, `double` | `real`, `double precision` | `float`, `double` | `real`, `float4`, `float8`, `double precision` | `real`, `float4`, `float8`, `double precision` | `float`, `real` | | `timestamp` | | `timestamp` | `timestamp` | `datetime`, `Datetime64` | `timestamp` | `Date`, `Timestamp` | `timestamp`, `timestamptz` | `timestamp` | `timestamp`, `timestamptz` | `timestamp(n)` | `timestamp`, `timestamptz` | `timestamp`, `timestamp_ntz`, `timestamp_tz`, `timestamp_ltz` | `datetime2` | | `date` | | `date` | `date` | `date` | `date` | N/A | `date` | `date` | `date` | `date` | `date` | `date` | `date` | | `json` | | `varchar`, `string` | `json` | `string`\* | `string` | `Object` | `json` | `json` | `json`, `jsonb` | `nchar`, `char`, `varchar2`, `nvarchar2` | `varchar(MAX)` | `varchar`, `variant` | `nvarchar(MAX)` | | `time` | | `time` | `time` | `time` | N/A | N/A | `time` | N/A | `time` | N/A | `time` | `time` | `time` | | `geography` | | `geometry` | `geography` | `Point`, `Ring`, `Linestring`, `Multilinestring`, `Polygon`, `Multipolygon` | N/A | `geojson` | `geometry` | `geometry`, `point`, `ring`, `linestring`, `multilinestring`, `polygon`, `multipolygon` | `geometry`, `point`, `ring`, `linestring`, `multilinestring`, `polygon`, `multipolygon` | N/A | `geometry`, `point`, `ring`, `linestring`, `multilinestring`, `polygon`, `multipolygon` | `geography`, `geometry` | `geography`, `geometry` | | `vector` | | `array` | `array` | `Array(Float64)` | `array` | `array(float)` | `double[]` | N/A | `double precision[]` | N/A | `array(double precision)` | `array(double precision)` | N/A | **Scale and Precision limitations for decimal types** In order to support broad compatibility across destinations, `decimal` typed values must fit within the `DECIMAL(38,19)` scale and precision. `decimal` values transferred to BigQuery will be truncated to `DECIMAL(38,9)`, as this is the maximum supported size. **Precision limitations for floating point types** Due to technical constraints, Prequel is unable to guarantee the absence of a loss of precision for any floating point (eg `float`) type. For use-cases in which precision is key, such as when transferring financial data, we **strongly recommend** leveraging fixed precision types instead such as `decimal`. **Geography type is limited to World Geodetic System** Every geometric shape has a spatial reference system associated with it. The `geography` type is limited to geospatial data that corresponds to an SRID of `4326`, which represents spatial data using longitude and latitude coordinates on the Earth's surface as defined in the WGS84 standard. **Choosing between `array`, `json`, and `vector`** * The `vector` Prequel type is used when syncing float-valued arrays (e.g., ML embeddings). * For all other array-like data, including text arrays, integer arrays, mixed types, or nested structures convert to `json` upstream and use the `json` Prequel data type. `json` has broader destination support and is the safer default for arbitrary arrays. **Timestamps are synced as epoch values** All `timestamp` values are stored as a UTC epoch (microseconds since Unix epoch), regardless of the source timezone. The source timezone has no bearing on what reaches the destination. How the destination displays that epoch depends on the destination system's timezone configuration. **Athena Iceberg limitations**: For Iceberg tables, Athena has two timestamp restrictions: * **Type**: Athena only supports `timestamp` (without time zone) in DDL operations. If your Athena Iceberg source has columns typed as `timestamp with time zone`, cast them to `timestamp` in your source before syncing. * **Precision**: Athena Iceberg only supports millisecond precision (`timestamp(3)`). If your source has `timestamp(6)` columns, cast them to `timestamp(3)` in your source before syncing. \* An asterisk indicates partial support. Ask us about any specific data type limitations. ## Destination type mapping If you are interested in what data type your recipient destination will receive, reference this table. | Prequel Type | Supported | Athena | BigQuery | Clickhouse | Databricks | Delta Lake | Apache Iceberg | MongoDB | MotherDuck | MySQL | Postgres | Oracle | Redshift | Snowflake | SQL Server | | ------------ | --------------------- | --------------- | ---------------- | ---------------- | ------------------ | --------------- | -------------------------- | -------------- | ----------------- | ----------- | -------------------- | -------------- | ------------------- | ------------------------- | ---------------------- | | `bytes` | | `string` | `bytes` | `blob` | `binary` | `varbinary` | `varbinary` | `binData` | `blob` | `binary` | `bytea` | `blob` | `varbyte(16777216)` | `binary` | `blob` | | `string` | | `string` | `string` | `string` | `string` | `varchar` | `varchar` | `string` | `text` | `text` | `text` | `clob` | `varchar(MAX)` | `text` | `nvarchar(MAX)` | | `boolean` | | `boolean` | `bool` | `boolean` | `boolean` | `boolean` | `boolean` | `boolean` | `boolean` | `boolean` | `boolean` | `number` | `boolean` | `boolean` | `bit` | | `integer` | | `integer` | `int64` | `integer` | `int` | `int` | `int` | `int` | `smallint`, `int` | `int` | `integer` | `int` | `bigint` | `integer` | `int` | | `bigint` | | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | `long` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | `bigint` | | `decimal` | | `decimal` | `decimal` | `decimal` | `decimal` | `decimal` | `decimal` | `decimal128` | `decimal` | `decimal` | `decimal` | `decimal` | `decimal` | `decimal` | `decimal` | | `float` | | `float` | `float64` | `double` | `double` | `double` | `double` | `double` | `double` | `double` | `double precision` | `double` | `double precision` | `double precision` | `float` | | `timestamp` | | `timestamp` | `timestamp` | `Datetime64` | `timestamp` | `timestamp(6)` | `timestamp with time zone` | `date` | `timestamptz` | `timestamp` | `timestamptz` | `timestamp(9)` | `timestamptz` | `timestamp_tz` | `datetime2` | | `date` | | `date` | `date` | `date` | `date` | `date` | `date` | `string` | `date` | `date` | `date` | `date` | `date` | `date` | `date` | | `json` | | `string` | `json` | `string`\* | `string` | `varchar` | `varchar` | `json` | `json` | `json` | `jsonb` | `clob` | `varchar(MAX)` | `variant` | `nvarchar(MAX)` | | `time` | | `time` | `time` | `time` | `time` | `varchar` | `varchar` | `string` | `time` | `time` | `time` | `varchar2(8)` | `varchar(MAX)` | `time` | `time` | | `geography` | | `geometry` | `geography` | `string` (wkt) | `string` (geojson) | `varchar` | `varchar` | `geojson` | `geometry` | `geometry` | `geometry` | `clob` | `geometry` | `geography` | `geography` | | `vector` | | `array` | `array` | `Array(Float64)` | `array` | `array(double)` | `array(double)` | `array(float)` | `double[]` | `json` | `double precision[]` | `clob` (json) | `super` | `array(double precision)` | `nvarchar(MAX)` (json) | **Delta Lake and Apache Iceberg types are Trino type names** Prequel creates Delta Lake and Apache Iceberg tables through Trino, so this table lists Trino type names. The engine you use to read the table reports the equivalent native type instead. For example, a `varchar` column in an Iceberg table is reported as `string` by Athena and Spark. \* An asterisk indicates partial or incomplete support. Ask us about any specific data type limitations. ### Object storage Parquet types Apache Parquet defines each column using a **Parquet type** (the physical storage format, such as INT64 or BYTE\_ARRAY) and optional type annotations: **convertedType** (legacy annotation) and **logicalType** (modern, richer metadata like TimestampType or DecimalType). Prequel selects these encodings to maximize compatibility across analytic engines (Athena, BigQuery, Spark, Trino, etc.) so your exported files load predictably without requiring engine-specific casts or configuration. | Prequel type | Parquet type | convertedType | logicalType | | ------------ | ---------------------- | ------------------ | ------------------------------------------------------------------------------------------------------ | | `bytes` | `BYTE_ARRAY` | `NULL` | `NULL` | | `string` | `BYTE_ARRAY` | `UTF8` | `NULL` | | `boolean` | `BOOLEAN` | `NULL` | `NULL` | | `integer` | `INT64` | `INT_64` | `NULL` | | `bigint` | `INT64` | `INT_64` | `NULL` | | `decimal` | `FIXED_LEN_BYTE_ARRAY` | `DECIMAL` | `DecimalType(precision=38, scale=19)` | | `float` | `DOUBLE` | `NULL` | `NULL` | | `timestamp` | `INT64` | `TIMESTAMP_MICROS` | `TimestampType(isAdjustedToUTC=1, unit=TimeUnit(MILLIS=, MICROS=MicrosSeconds(), NANOS=))` | | `date` | `BYTE_ARRAY` | `UTF8` | `NULL` | | `json` | `BYTE_ARRAY` | `UTF8` | `NULL` | | `time` | `BYTE_ARRAY` | `UTF8` | `NULL` | # Destinations Source: https://docs.prequel.co/export/concepts/destinations Understanding how Prequel writes data to your customers' destinations A **destination** is the database, data warehouse, object storage bucket, or spreadsheet where a recipient's data is written. Each destination belongs to exactly one [recipient](/export/concepts/recipients). For vendor-specific setup instructions and a full list of supported destination types, see the [Destination Guides](/export/destinations/overview). # Environments Source: https://docs.prequel.co/export/concepts/environments Learn about the built-in Prequel environments ## What are the Prequel environments? Every Prequel account is provisioned with two environments: `staging` and `prod`. Functionally, they behave the same, but with a few key differences. | Considerations by Environment | `prod` | `staging` | | :---------------------------- | :--------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Support** | Standard SLA as guaranteed in Service Agreement. | Prequel SLAs do not apply to `staging` environments. Tactically, this means Prequel support is not monitoring `staging` environments, and will not actively detect or respond to issues. | | **Billing** | Billed according to your Service Agreement. | Prequel does not bill for `staging` usage. | | **Product** | Schema evolution considerations mean column data types cannot be modified (only added/removed) | Schema evolution features are not enabled. Modifying column data types may break transfers until a `full_refresh` is performed. | ## Switching between `staging` and `prod` environments ### API keys: `sk_staging_*` vs. `sk_prod_*` API keys beginning with `sk_staging_` will automatically route your request to your `staging` environment. API keys beginning with `sk_prod_` will route requests to `prod`. ### Admin UI toggle In the Admin UI, the toggle at the bottom left of the screen can be used to switch between or verify environments. ![](https://storage.googleapis.com/prequel_docs/images/ui-admin-environments-toggle.png) # Models Source: https://docs.prequel.co/export/concepts/models Understanding Prequel models ## What are data models? In this documentation, a **data model** refers to the Prequel representation of the structured data to be transferred using Prequel. An individual data model usually represents a single **table**, though (depending on the source/destination) it may also refer to a **JSON** blob or other **file** format (e.g., `Parquet`, `CSV`). ![](https://storage.googleapis.com/prequel_docs/images/prequel-data-models-export.png "A Prequel data model represents the data structure that is read from a source and written to a destination") ## Data model configuration For the complete list of fields, their editability, and environment-specific constraints, see the [Update Model endpoint](/export/api-reference/models/update-model) in the API reference. Each Prequel data model is configured via a JSON file with a specific format. If no `model_name` is provided, the name of the file (not including the extension) will be the name used for the table in the destination database. For example, for the configuration below, the destination table will appear as `logs`. ```json title="logs.json" theme={null} { "model": { "model_name": "logs", "columns": [ { "name_in_source": "id", "name_in_destination": "id", "data_type": "text", "is_primary_key": true }, { "name_in_source": "log", "name_in_destination": "event_log", "description": "A descriptive text entry of the event that occurred.", "data_type": "text" }, { "name_in_source": "updated_at", "name_in_destination": "updated_at", "data_type": "timestamp", "is_last_modified": true } ], "source_table": "source_schema.application_logs", "source_name": "Example Production Source", "organization_column": "organization_id" } } ``` **Syncing config files from GitHub** You can use the [Prequel GitHub Action](/export/integrations/github-action) to automatically sync any config file changes from GitHub to Prequel. ### Required columns Every table to be transferred must expose the following columns in its source so Prequel can move and reconcile rows correctly: | Required column | Configured via | Why it is required | | :---------------------------------- | :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Unique ID (e.g., `id`) | `is_primary_key` on a column | Used as the primary key to facilitate `UPDATE`/`INSERT` ("upsert") operations on the destination. | | Last modified (e.g., `updated_at`) | `is_last_modified` on a column | A `timestamp` column indicating when a row last changed. Prequel uses it to identify changes between transfers. | | Tenant ID (e.g., `organization_id`) | `organization_column` | Required in **multi-tenant tables** mode to filter source data by tenant. Not required in **schema-tenanted** mode. See [Tenancy options](/export/concepts/multi-tenancy). | ### Configuration file keys Each configuration file has 7 top level config keys: `columns` and `source_table` are always required; `organization_column` is required in multi-tenant tables mode; `model_name`, `description`, `source_name`, and `source_id` are optional. #### Model config fields Name of the table to be loaded into the destination. If left blank, the file name (not including the file extension) is used. Table description. Applied to destination tables if provided. A list of all destination columns for this table. This is used to `CREATE` the table in the destination database, so use accurate types that correspond to the source data. The order of this list dictates the order in which columns appear in the destination database. These columns also build the `READ` and `WRITE` queries, so use the correct `name_in_source` and a descriptive `name_in_destination` for end customers. They can, but do not need to, match. See [Column fields](#column-fields) for the per-column keys. The schema-qualified name of the source table to be queried, in the form `schema.table`. In **schema-tenanted** mode, the schema qualifier is used only for [model validation](#model-validation). At transfer time, Prequel uses each recipient's `id_in_provider_system` as the schema instead. See [Tenancy options](/export/concepts/multi-tenancy). Associates this model with a source by name. Required when multiple sources exist. If omitted with a single source, that source is used automatically. If provided, the value must match an existing source name, even in single-source accounts. Associates this model with a source by ID. Carries the same enforcement as `source_name`. Both fields can be specified together, in which case the source must match both values. Required in **multi-tenant tables** mode. Specifies the column in the source table used to filter rows by tenant. Not required in **schema-tenanted** mode, where tenancy is determined by schema name instead; if supplied, the value is ignored. Because this column is not necessarily one you want to transfer to recipients, it does not need to appear in the `columns` list. Must be a string or `varchar` (non-numeric) data type. **Choosing between `source_name` and `source_id`** Prefer `source_name` for config files shared across staging and production environments. Source names are consistent across environments, whereas source IDs are environment-specific UUIDs that differ between deployments. Use `source_id` when you need to reference a source by its UUID, or specify both fields together to require the source to match on both name and ID. #### Model validation When you create or update a model with [`POST /export/models`](/export/api-reference/models/create-model) or [`PATCH /export/models/{model_id}`](/export/api-reference/models/update-model), Prequel automatically validates the model against the source to ensure all columns exist, and both the source types and sampled source values match the model's types. Validation also runs the model's `source_query` (or generated table query) against the source database, checks the configured columns against the result set, and verifies that the source credentials have sufficient access. You can also re-run validation on demand against an existing model with [`POST /export/models/{model_id}/validate`](/export/api-reference/models/validate-model). To skip validation on a single create or update request, set `skip_model_validation` to `true` in the request body. Skipping validation does not affect transfers. The following optional model config fields let you supply concrete values that Prequel uses **only during validation**. Use them to point validation at a known tenant and a narrow time window to improve performance of the validation query. These parameters are ignored by transfers at runtime. Set to `true` on a single create or update request to skip validation. Skipping validation does not affect transfers. Overrides `{{.IdInProviderSystem}}` during validation. RFC3339 timestamp. Overrides `{{.LastModifiedAtEpoch}}` during validation. RFC3339 timestamp. Overrides `{{.LastModifiedAtEpochUpperBound}}` during validation. ```json title="Validation overrides" icon="brackets-curly" expandable theme={null} { "model": { "model_name": "events", "source_query": "SELECT * FROM events WHERE tenant_id = '{{.IdInProviderSystem}}' AND updated_at >= to_timestamp({{.LastModifiedAtEpoch}}){{with .LastModifiedAtEpochUpperBound}} AND updated_at < to_timestamp({{.}}){{end}}", "model_validation_id_in_provider_system": "tenant-abc", "model_validation_start_time_epoch": "2025-01-01T00:00:00Z", "model_validation_end_time_epoch": "2025-01-02T00:00:00Z", ... } } ``` #### Column fields Each entry in the `columns` array accepts these keys. The column name in the source system. The column name to use in the destination table. The logical data type for this column. Refer to [Acceptable Types](/export/concepts/data-types). Marks the primary key column used to identify rows. Exactly one column must set this. Marks the column containing the last-updated timestamp used for change detection. Exactly one column must set this. Optional human-readable description for the column. **`is_primary_key` and `is_last_modified` must appear within the first 32 columns** Both the primary key column and the last modified column must be positioned within the first 32 entries of the `columns` array (positions 1–32). This is a Prequel model requirement. It does not constrain the column order in your source table. The limit exists to respect ordinal position constraints in destination systems that implement indexing or partitioning by specific columns. **Using Acceptable Types** Use the correct `data_type` for each column to maintain compatibility with all destinations. Refer to the [Data Types](/export/concepts/data-types) documentation for the complete list of supported types. #### Column ordering The order of columns in the `columns` array determines the column order in the destination table, but only at initial table creation. For existing destinations, columns added via schema evolution are always appended to the end of the destination table schema, regardless of their position in the model config. See [Schema evolution](/export/features/schema-evolution) for more detail. #### Column name and table name conflicts A column's `name_in_source` and `name_in_destination` cannot match the source table name or the destination table name (case-insensitive). This applies to both the column name as written and the table name as written. #### Reserved column names Some reserved keywords cannot be used as column names due to reserved statuses in common destination systems. There are 334: | 1-67 | 68-134 | 135-201 | 202-268 | 269-334 | | :--------------------- | :----------------- | :------------------------------ | :------------------- | :-------------------- | | `ACCESSIBLE` | `DEFERRABLE` | `INOUT` | `NULL` | `SMALLINT` | | `ACCOUNT` | `DEFINE` | `INSENSITIVE` | `NULLS` | `SOME` | | `ADD` | `DELAYED` | `INSERT` | `NUMERIC` | `SPATIAL` | | `ALL` | `DELETE` | `INT` | `OF` | `SPECIFIC` | | `ALTER` | `DENSE_RANK` | `INT1` | `OFFSET` | `SQL` | | `ANALYSE` | `DESC` | `INT2` | `ON` | `SQL_BIG_RESULT` | | `ANALYZE` | `DESCRIBE` | `INT3` | `ONLY` | `SQL_CALC_FOUND_ROWS` | | `AND` | `DETERMINISTIC` | `INT4` | `OPTIMIZE` | `SQL_SMALL_RESULT` | | `ANTI` | `DISTINCT` | `INT8` | `OPTIMIZER_COSTS` | `SQLEXCEPTION` | | `ANY` | `DISTINCTROW` | `INTEGER` | `OPTION` | `SQLSTATE` | | `ARRAY` | `DIV` | `INTERSECT` | `OPTIONALLY` | `SQLWARNING` | | `AS` | `DO` | `INTERVAL` | `OR` | `SSL` | | `ASC` | `DOUBLE` | `INTO` | `ORDER` | `START` | | `ASENSITIVE` | `DROP` | `IO_AFTER_GTIDS` | `ORGANIZATION` | `STARTING` | | `ASSERT_ROWS_MODIFIED` | `DUAL` | `IO_BEFORE_GTIDS` | `OUT` | `STORED` | | `ASYMMETRIC` | `EACH` | `IS` | `OUTER` | `STRAIGHT_JOIN` | | `AT` | `ELSE` | `ISNULL` | `OUTFILE` | `STRUCT` | | `AUTHORIZATION` | `ELSEIF` | `ISSUE` | `OVER` | `SYMMETRIC` | | `BEFORE` | `EMPTY` | `ITERATE` | `OVERLAPS` | `SYSTEM` | | `BETWEEN` | `ENCLOSED` | `JOIN` | `PARTITION` | `TABLE` | | `BIGINT` | `END` | `JSON_TABLE` | `PERCENT_RANK` | `TABLESAMPLE` | | `BINARY` | `ENUM` | `KEY` | `PLACING` | `TERMINATED` | | `BLOB` | `ESCAPE` | `KEYS` | `PRECEDING` | `THEN` | | `BOTH` | `ESCAPED` | `KILL` | `PRECISION` | `TINYBLOB` | | `BY` | `EXCEPT` | `LAG` | `PRIMARY` | `TINYINT` | | `CALL` | `EXCLUDE` | `LAST_VALUE` | `PROCEDURE` | `TINYTEXT` | | `CASCADE` | `EXISTS` | `LATERAL` | `PROTO` | `TO` | | `CASE` | `EXIT` | `LEAD` | `PURGE` | `TRAILING` | | `CAST` | `EXPLAIN` | `LEADING` | `QUALIFY` | `TREAT` | | `CHANGE` | `EXTRACT` | `LEAVE` | `RANGE` | `TRIGGER` | | `CHAR` | `FALSE` | `LEFT` | `RANK` | `TRUE` | | `CHARACTER` | `FETCH` | `LIKE` | `READ` | `UNBOUNDED` | | `CHECK` | `FIRST_VALUE` | `LIMIT` | `READ_WRITE` | `UNDO` | | `COLLATE` | `FLOAT` | `LINEAR` | `READS` | `UNION` | | `COLLATION` | `FLOAT4` | `LINES` | `REAL` | `UNIQUE` | | `COLUMN` | `FLOAT8` | `LOAD` | `RECURSIVE` | `UNLOCK` | | `CONCURRENTLY` | `FOLLOWING` | `LOCALTIME` | `REFERENCES` | `UNNEST` | | `CONDITION` | `FOR` | `LOCALTIMESTAMP` | `REGEXP` | `UNSIGNED` | | `CONNECT` | `FORCE` | `LOCK` | `RELEASE` | `UPDATE` | | `CONNECTION` | `FOREIGN` | `LONG` | `RENAME` | `USAGE` | | `CONSTRAINT` | `FREEZE` | `LONGBLOB` | `REPEAT` | `USE` | | `CONTAINS` | `FROM` | `LONGTEXT` | `REPLACE` | `USER` | | `CONTINUE` | `FULL` | `LOOKUP` | `REQUIRE` | `USING` | | `CONVERT` | `FULLTEXT` | `LOOP` | `RESIGNAL` | `UTC_DATE` | | `CREATE` | `FUNCTION` | `LOW_PRIORITY` | `RESTRICT` | `UTC_TIME` | | `CROSS` | `GENERATED` | `MASTER_BIND` | `RETURN` | `UTC_TIMESTAMP` | | `CUBE` | `GET` | `MASTER_SSL_VERIFY_SERVER_CERT` | `RETURNING` | `VALUES` | | `CUME_DIST` | `GRANT` | `MATCH` | `REVOKE` | `VARBINARY` | | `CURRENT` | `GROUP` | `MAXVALUE` | `RIGHT` | `VARCHAR` | | `CURRENT_CATALOG` | `GROUPING` | `MEDIUMBLOB` | `RLIKE` | `VARCHARACTER` | | `CURRENT_DATE` | `GROUPS` | `MEDIUMINT` | `ROLLUP` | `VARIADIC` | | `CURRENT_ROLE` | `GSCLUSTER` | `MEDIUMTEXT` | `ROW` | `VARYING` | | `CURRENT_SCHEMA` | `HASH` | `MERGE` | `ROW_NUMBER` | `VERBOSE` | | `CURRENT_TIME` | `HAVING` | `MIDDLEINT` | `ROWS` | `VIEW` | | `CURRENT_TIMESTAMP` | `HIGH_PRIORITY` | `MINUS` | `SAMPLE` | `VIRTUAL` | | `CURRENT_USER` | `HOUR_MICROSECOND` | `MINUTE_MICROSECOND` | `SCHEMA` | `WHEN` | | `CURSOR` | `HOUR_MINUTE` | `MINUTE_SECOND` | `SCHEMAS` | `WHENEVER` | | `DATABASE` | `HOUR_SECOND` | `MOD` | `SECOND_MICROSECOND` | `WHERE` | | `DATABASES` | `IF` | `MODIFIES` | `SELECT` | `WHILE` | | `DAY_HOUR` | `IGNORE` | `NATURAL` | `SEMI` | `WINDOW` | | `DAY_MICROSECOND` | `ILIKE` | `NEW` | `SENSITIVE` | `WITH` | | `DAY_MINUTE` | `IN` | `NO` | `SEPARATOR` | `WITHIN` | | `DAY_SECOND` | `INCREMENT` | `NO_WRITE_TO_BINLOG` | `SESSION_USER` | `WRITE` | | `DEC` | `INDEX` | `NOT` | `SET` | `XOR` | | `DECIMAL` | `INFILE` | `NOTNULL` | `SHOW` | `YEAR_MONTH` | | `DECLARE` | `INITIALLY` | `NTH_VALUE` | `SIGNAL` | `ZEROFILL` | | `DEFAULT` | `INNER` | `NTILE` | `SIMILAR` | \`\` | #### Reserved column name prefixes In order to avoid conflicts in common destination systems, column names cannot start with certain reserved prefixes. There are 10 reserved prefixes: | 1-5 | 6-10 | | :--------------- | :------------------------ | | `PARGO_PREFIX_` | `_TABLE_` | | `_FILE_` | `_PARTITION` | | `_ROW_TIMESTAMP` | `__ROOT__` | | `_COLIDENTIFIER` | `_CHANGE_SEQUENCE_NUMBER` | | `_CHANGE_TYPE` | `_CHANGE_TIMESTAMP` | ## Grouping data models into products By default, every destination you add to Prequel will be sent all configured models. **Products** allow you to only send a subset of those models to destinations. # Tenancy options Source: https://docs.prequel.co/export/concepts/multi-tenancy Setting your multi-tenancy mode ## What is multi-tenancy? Multi-tenancy refers to storing data from multiple users of your software (tenants) in a single database. Prequel is purpose-built to sync data from, and to, multi-tenanted data stores. Tenancy is commonly modeled in one of two ways. * **Multi-tenant tables**, where tenant data is commingled in a shared set of tables, with a column to indicate tenancy (for example, an `organization_id` or `tenant_id` column). * **Schema per tenant**, where each schema contains the data for a single tenant. ![](https://storage.googleapis.com/prequel_docs/images/prequel-table-vs-schema-tenant.png) | Multi-tenant mode | Description | | :---------------------------------- | :---------------------------------------------------------------------------------------------------------- | | **Multi-tenant tables** (`default`) | All tenants occupy the same tables, and tenancy is indicated via a column (for example, `organization_id`). | | **Schema-tenanted database** | All tenants occupy their own schema, and tenancy is indicated by the schema. | ## How do I indicate my multi-tenancy mode? **Toggling your data tenancy mode** By default, every Prequel account is provisioned in **multi-tenant tables** mode. To switch your account to **schema-tenanted database** mode, contact Prequel support. In multi-tenant tables mode, set each model's `organization_column` to the column that identifies the tenant, and Prequel filters every transfer to the receiving recipient's `id_in_provider_system`. See [Models](/export/concepts/models) and [Recipients](/export/concepts/recipients) for how the tenant identifier flows through a transfer. ## What differs in schema-tenanted mode? All features are available in both multi-tenant tables and schema-tenanted modes. The exact differences between how you use each mode are listed below. | Behavior | Multi-tenant tables (`default`) | Schema-tenanted database | | :-------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------- | | Tenant filtering | Prequel filters every transfer to rows where the model's `organization_column` matches the recipient's `id_in_provider_system`. | No row filter is applied. Each transfer reads the tenant's tables from the recipient's schema in your source. | | `organization_column` | Required on every model. | Not required. If supplied, the value is ignored. | | Schema in a model's `source_table` field | Used for every transfer. | Used only in model validation at model creation or update time. At transfer time, Prequel replaces it with the recipient's `id_in_provider_system`. | | Recipient `id_in_provider_system` | The tenant identifier matched against the `organization_column` values. | The tenant's exact schema name in your source. | | [Source query](/export/features/custom-queries) variables | `{{.IdInProviderSystem}}` and `{{.OrganizationColumnName}}` are available. | `{{.Schema}}` is available, and resolves to the recipient's `id_in_provider_system`. | # Products Source: https://docs.prequel.co/export/concepts/products Understanding Products, an optional Prequel feature ## What are products? By default, every recipient added to Prequel has access to all models defined in the `prequel/models` directory. **Products** is an optional feature that allows you to subscribe recipients to a subset of available models. ![](https://storage.googleapis.com/prequel_docs/images/prequel-recipients-and-products.png "Every **Product** is a list of Models, and every Recipient can subscribe to a list of Products") **Products - an example use case** Say you operate a SaaS business with 3 product suites: "Sales Tools", "Marketing Tools", and "Customer Support Tools". Any given product suite may have a different set of underlying tables, but a given customer can subscribe to 1, 2, or even all of them. You might use the Prequel Products feature to group these tables into "Sales Tools Dataset", "Marketing Tools Dataset", and "Customer Tools Dataset", so that based on the given recipient, you can make a different set of tables available to them. ## Getting started with products Create a new directory in your git repo called `prequel/products`. In this directory, create a `default.json` file. This file specifies which models are sent to a destination if you don't add them to any specific products. In other words, it specifies which models are sent to destinations by default. Like all product files, it takes a single key, `models`, and the corresponding value should be a list of model names. Let's imagine we have three models defined in our `prequel/models` directory: `accounts.json`, `logs.json`, and `transactions.json`. Let's also imagine we define the following `default.json` file in `prequel/products`. ```json title="default.json" theme={null} { "models": ["accounts", "logs"] } ``` Now, destinations added without a `products` field will only receive the `accounts` and `logs` models, and will not receive the `transactions` model. **Validate product files** Product files can be validated in the same way that model files are. Simply make an API call to the `/products/{product_name}/validate` endpoint. For example: ```bash title="Validate product" icon="terminal" theme={null} curl https://api.prequel.co/products/default/validate -X POST -d '' ... ``` ## Non-default products For the sake of example, let's assume that we still want certain destinations to receive the `transactions` model. We can define a new product -- we'll name it `transactions_feature`. ```json title="transactions_product.prql" theme={null} { "models": ["transactions"] } ``` Specific destinations can now be configured with both products -- `default` and `transactions_feature` -- and they will receive the default tables, as well as the transaction table (`accounts`, `logs`, `transactions`) while destinations who aren't assigned to a specific product will only receive `accounts` and `logs`. ## Assigning destinations to products By default, destinations are assigned to the `default` product. In other words, if you take no action when you create a destination, they will receive all the models defined in the `default.json` product. # Recipients Source: https://docs.prequel.co/export/concepts/recipients Understanding Recipients **Recipients** represent a unique entity you are exporting data to with Prequel. Recipients most often represent a customer, but there may be use cases to represent a recipient as a unique tenant within a customer account, such as a project. For more on tenanting strategy with Recipients, continue reading below. Every Prequel Destination is required to have a Recipient, which defines what data models and product(s) a customer should receive. ![](https://storage.googleapis.com/prequel_docs/images/prequel-recipients-and-products.png) ### What is `id_in_provider_system` `id_in_provider_system` refers to the recipient's identifier by which data will be filtered per organization. In other words, it is the link between a Prequel Recipient and the corresponding identifier in the data source. In multi-tenant tables mode, the value assigned to this key will be matched against the values under the `organization_id` column provided for every table. In schema-tenanted mode, the value must be the tenant's exact schema name in the source, and each transfer reads the recipient's data from that schema. ### Resolving multiple recipients and destinations Prequel enforces a key rules on uniqueness of Recipients and Destinations: * Multiple Recipients cannot point to the same Destination. Prequel's [change detection](/export/features/change-detection) logic relies on guarantees of a destination's state on any given transfer, such that writing multiple recipients' data to the same table can cause non-deterministic transfer behavior. * To write data from multiple recipients to the same database or warehouse, it is required that one Prequel Destination is defined for each Recipient, where **each Destination must have a unique schema** to avoid collisions. * Each `id_in_provider_system` must be unique. ### Using schema-based tenanting Prequel supports schema-based tenanting. To add a destination using schema-based tenanting, first configure a `recipient` whose `id_in_provider_system` is set to the tenant's schema name in the source, then create the destination. Read more about multi-tenancy in Prequel here: [Multi-tenancy](/export/concepts/multi-tenancy). ### Updating products for a given destination When adding a Destination for a given Recipient for the first time, Prequel provides a helpful shortcut to abstract away the need to create the Recipient beforehand (that is, if a Destination is created with a `id_in_provider_system` that has not been used, the Recipient with that `id_in_provider_system` is created behind the scenes). However, when Products need to be updated for a Recipient, that update must happen on the Recipient object, using the `PATCH /recipients/{recipient_id}` endpoint. ### Using role-based access control on a source Prequel supports using specific source users for a given recipient's data. To add a specific set of credentials that should be used to access source data for a given recipient, you can use the `POST /recipient-source-credentials` [endpoint](/export/api-reference/recipient-source-credentials/create-source-credential-for-a-recipient). # Sources Source: https://docs.prequel.co/export/concepts/sources Understanding how Prequel connects to your source data "Sources" refers to the databases or data warehouses from which data will be transferred. Prequel supports many different sources: you can find the exhaustive list of supported `vendor` values on the [Create source endpoint](/export/api-reference/sources/create-source). ## Connecting your source Prequel requires different permissions and configurations depending on your source. For a step-by-step guide on how to configure and add your source to Prequel, select the source you're using from the left-hand menu under the Sources heading. ## Multiple sources By default, Prequel works with a single source. When set up in that way, all models pull data from the single available source. However, there are cases where you might want to add multiple sources. For example, you may want to sync certain tables from your data warehouse, and a few other tables from a Postgres database running elsewhere. To facilitate those use-cases, Prequel can be run in multiple sources mode. In a multi-source setup, models need to specify which source they intend to read data from. For example, model `foo.prql` might read from source A, and `bar.prql` might read from source B. This is done by adding a `source_name` (or `source_id`) field to each model. The value should be the name (or ID) of the relevant source. If multiple sources exist and a model does not have a `source_name` or `source_id`, creating or updating that model will return a validation error. As such, it's important to add the `source_name` field to your models before adding more than one source. Checklist for switching to a multiple sources setup. * Add `source_name` (or `source_id`) to existing models. * Add a new source through the [API endpoint](/export/api-reference/overview). * Add new models that point to the new source. # Transfers Source: https://docs.prequel.co/export/concepts/transfers Understanding the Prequel data transfer logic ## How transfers work Prequel performs transfers by querying the source for a given recipient's data and loading that data into the recipient's destination, on an ongoing basis. The first transfer that runs for a given destination will automatically load all historical data (the "backfill"), and subsequent transfers will attempt to transfer only the data that has changed or been added since the previous transfer. ### Prequel transfers from source to destination Prequel authenticates to [Sources](/export/concepts/sources) using scoped credentials or delegated roles created by the user. Prequel validates connectivity, and restricts permissions to only what is needed to read the configured models for the intended recipient for **least-privilege access with clear auditability.** Data is read in a sliding window based on time. Each transfer moves a window of data, starting from a checkpoint based on the last batch of data transferred to ensure **data integrity and efficient transfers at scale**. When available, Prequel uses a source staging bucket to temporarily store the results of queries as files in object storage which are then downloaded and normalized. Prequel uses a lookback window to ensure resiliency against **eventual consistency** concerns in data sources. For more detail on its mechanics, see [Change detection](/export/features/change-detection#eventual-consistency). Prequel authenticates to your customer's destinations using destination-native authentication scoped to the target schemas/tables for **isolation and least-privilege access aligned with destination security.** * **Staging-assisted loads**: Batches are uploaded to a staging area (for example, a native volume or storage bucket) and then ingested using the destination's bulk-load path. Data is normalized before staging. Prequel's transfer logic is designed uniquely for each destination type to **maximize throughput and leverage vendor-optimized patterns.** * **Direct inserts**: For destinations that don't support staging-assisted loads, batches are streamed directly via insert SQL queries or API calls without external staging. As a result, these destination types can have throughput limitations; contact the Prequel team to learn more about data volumes and throughput across destination types. * Prequel uses upserts with changes matched on primary key and duplicates resolved via the last modified timestamp to ensure **data integrity and protect table state**. * With a [Write-Ahead-Publish](https://lakefs.io/blog/data-engineering-patterns-write-audit-publish/) architecture, your customer never sees data before a transfer is complete and all data is available in the destination. * Staging files created during transfers are **automatically cleaned up after transfer completion**. Data is not persisted in the staging area after transfer. * With each transfer, metadata is written to each destination per transfer. For object storage locations, see [Manifest files for object storage](/export/features/manifest-files-for-object-storage), and for warehouses and databases, see [Transfer status table](/export/features/transfer-status-table). Each phase emits structured [logs](/export/monitoring/monitoring) and [metrics](/export/features/usage-data) for **governance and auditability**. Tags can be used to label transfers for filtering and reporting. ## Transfer lifecycle Transfers are managed by an internal queue, which is used to dispatch transfers to workers. When a destination has the `enabled` flag set to `true`, Prequel will automatically enqueue transfers for that destination based on the `frequency` value of the destination or the organization's default frequency. A transfer resource always has a status corresponding to its current phase of the lifecycle: | Status | Description | | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PENDING` | Transfers start as pending when they are created (enqueued). The `submitted_at` timestamp records when the transfer was enqueued. | | `RUNNING` | The transfer has been dispatched to a worker. The `started_at` timestamp records when it changed to `RUNNING`. | | `ERROR` | There was an issue dispatching the transfer, the worker failed to connect to the source or destination, or all models failed to transfer. `ended_at` records the change to `ERROR`. | | `PARTIAL_FAILURE` | The transfer reached the running state, but only some models succeeded while others failed. `ended_at` records the change to `PARTIAL_FAILURE`. | | `SUCCESS` | The transfer was running and all models transferred without issues. `ended_at` records the change to `SUCCESS`. | | `CANCELLED` | A user terminated the transfer before it started running. | | `KILLED` | A user terminated the transfer while it was running. | | `EXPIRED` | The transfer was blocked from being dispatched and remained pending for longer than 6 hours. | | `ORPHANED` | The worker died ungracefully or stopped communicating with the control plane. | ## Backfills & full refreshes The initial transfer (or "backfill"), is often the largest transfer by volume. During this initial sync, all historical data for a given recipient is loaded into the destination. To trigger a full refresh manually, add `"full_refresh": true` to a [transfer request](/export/api-reference/transfers/create-transfer). Prequel only triggers a full refresh automatically on the first transfer, either to a new destination or a new model. **Data impact varies by destination type** **Warehouses, databases, and open table format (OLAP, OLTP, OTF):** All existing data is deleted before reloading. If your source retains only partial history (e.g., a 90-day rolling window), **data outside that range will be lost**. Any date filters explicitly configured in a model's source query **also still apply**. **Object storage (non-OTF) & SFTP:** Existing files are not deleted, and a full refresh will produce duplicate data. **Backfill vs. incremental transfer performance** Because the initial backfill is often the most storage and compute intensive, sync time/performance should not be used as an indicator of ongoing transfer statistics. **Table Reset Behavior:** For warehouse and database destinations, Prequel determines whether to truncate or drop and recreate the table based on schema compatibility: * **Truncate:** If the schema matches, the table is truncated before reloading data. * **Drop & Recreate:** If there is a schema mismatch, the table is dropped and recreated with the correct schema. - A customer accidentally drops or overwrites one or more tables in their destination system. - A new column was added and historical data needs to be backfilled. * **Transfer ended in a non-`SUCCESS` status (`ERROR`, `PARTIAL_FAILURE`, `ORPHANED`):** Prequel tracks sync checkpoints per-model. Any model that did not complete successfully will automatically resume from its last successful checkpoint during the next incremental transfer. * **Adding a new model:** Prequel triggers the initial backfill automatically. * **Non-persisted data in source:** If your source does not preserve full customer data history, a full refresh transfer will result in data loss. If you're unsure whether a full refresh fits your situation, contact Prequel support to discuss your use case. ## Incremental transfers After each transfer (backfill or incremental) Prequel will record the most recent last modified timestamp value transferred. This value will be used as the starting point for the subsequent transfer. By default, every transfer of a given model (after a successful backfill) will be an "incremental transfer". **Incremental updates and eventually consistent data sources** By default, Prequel will query the source for slightly earlier data than the most recently transferred row. This is to provide a window in which data from eventually consistent sources can converge and still be transferred. ## Transfer parallelism and concurrency **Transfer Concurrency:** Within an individual transfer, operations are optimistically concurrent. Transfers can download, upload, or serialize multiple data files concurrently, regardless of the model to which they belong. The `max_concurrent_queries_per_transfer` field on a [source](/export/api-reference/sources/create-source) or [destination](/export/api-reference/destinations/create-destination) limits the number of concurrently queries or API calls that can be made against the source or destination. The default for `max_concurrent_queries_per_transfer` is `1`. **Transfers Parallelism:** Transfers can run in parallel of each other as long as the following constraints hold: * No simultaneous transfers are allowed for the same model to the same destination. * No simultaneous integrity and transfer jobs can run against the same destination. * The `max_concurrent_transfers` field exists on both the source and destination. It defaults to `10` for sources and `1` for destinations. This field represents a hard limit on the number of simultaneous transfers involving a particular source or destination. Prequel's dispatcher will enforce the above rules. A transfer that is unable to be dispatched will remain pending until it can be dispatched. ## Tags Every transfer can carry an arbitrary set of **tags**, simple key/value metadata you define to group, label, or annotate transfers (for example, by environment, team, workload, etc.). Tag keys and values must match `^[A-Za-z0-9_-]+$`. For more detail on how to use Tags when [creating transfers](/export/api-reference/transfers/create-transfer) and [filtering transfers](/export/api-reference/transfers/list-transfers) on tags, refer to our API Reference. ## Staging buckets Some sources and destinations supported by Prequel may require staging buckets to efficiently transfer data. Where possible, Prequel will use built in staging resourced provided by the database or data warehouse, but in cases where it does not exist, it may need to be provided. The source/destination documentation will provide instructions for configuring staging buckets where needed. # Deployment options Source: https://docs.prequel.co/export/deployment/cloud-hosted Choosing your preferred Prequel deployment option There are three options for deploying Prequel ## Cloud hosted If you choose Cloud Hosted, you will receive an onboarding email with instructions for signing into Prequel as soon as your cloud hosted deployment is ready. ## Private-cloud hosted In a private-cloud deployment, the Prequel team will be right there with you to ensure a smooth delivery. You will receive an onboarding email with instructions for signing into the shared-cloud instance of Prequel as soon as it's ready. ## Self-hosted Deploy Prequel on cloud infrastructure you control. Select your cloud below for the full guide. Self-hosted deployments rely on a few tools: * **Terraform v1.0.x** for provisioning the services required by Prequel * **Helm 3.9.4+** for installing and upgrading Prequel * **Kubernetes CLI** for managing and inspecting the Kubernetes cluster - Validate that you have access to the Terraform directory and Helm chart we sent over. If not, please email or Slack `support@prequel.co` to request access. - Create the dedicated cloud project where you'd like Prequel to run. We typically recommend creating a new cloud project for this, which allows all resources to be fully sandboxed from any other existing infrastructure and ensures that there will not be contention between VPCs, networks, or other resources. In the project you created for Prequel, navigate to the AWS Certificate Manager and `Request a certificate`. We'll need a certificate here for `*.your-domain.com`, since this is how we'll enable TLS/HTTPS for the Prequel deployment. Grab the ARN of the certificate you created and keep it handy for later. Take a look through `variables.tf` and fill in the required values. We have a `terraform.tfvars.example` that can be your reference. Perform a Terraform dry-run and double check that everything looks good. ```hcl title="Terraform" theme={null} terraform plan ``` Terraform the `main.tf` file. This will create all the necessary infrastructure for Prequel to run. Save the output variables, you'll need them later. ```hcl title="Terraform" theme={null} terraform apply ``` Auth into the cluster we just created. ```shell title="Authenticate to cluster" icon="terminal" theme={null} aws eks update-kubeconfig --name clustername --region clusterregion ``` **K8s cluster access** Once you've setup the Prequel infrastructure and connected to the EKS cluster, we highly recommend adding additional users/groups to the access entries of the cluster. This will ensure that if the original creator of the cluster loses access to the cluster, there will be additional users who can access and manage the cluster. This can be done via UI or via terraform. If you want to do it through terraform, you can leverage the `access_entries` variable, and there is an example in the `terraform.tfvars.example` file. Accessing and managing the cluster is necessary to support the deployment, maintain Prequel and update the Prequel software. Please see AWS documentation below with information how to add users/groups to the access entries. [https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html](https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html) Once the users/groups have been added to the access entries, you can instruct the new users to use the instructions below to create a new kube config to access the cluster. [https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html) Create the following Kubernetes secrets that Prequel requires: ```shell title="Create Kubernetes secrets" icon="terminal" expandable theme={null} # Generate and store secure random values in environment variables export POSTGRES_PASSWORD={your_db_password} export WORKOS_API_KEY={workos_api_key} export SSH_SALT={your_generated_ssh_salt} export ADMIN_API_KEY={your_generated_admin_api_key} export AUTH_TOKEN_KEY={your_generated_auth_token_key} # Store the keys provided to you by Prequel export LICENSE_KEY={license_key_from_prequel} export HONEYCOMB_API_KEY={honeycomb_api_key_from_prequel} # Create secret for Postgres DB credentials kubectl create secret generic datafeed-postgres \ --from-literal=password="${POSTGRES_PASSWORD}" # Create secret for SSH salt (used for hashing public keys) kubectl create secret generic datafeed-ssh-salt \ --from-literal=salt="${SSH_SALT}" # Create secret for Shepherd service kubectl create secret generic datafeed-shepherd \ --from-literal=apiKey="${ADMIN_API_KEY}" \ --from-literal=authToken="${AUTH_TOKEN_KEY}" \ --from-literal=workOSApiKey="${WORKOS_API_KEY}" # Create secret for the Prequel license key kubectl create secret generic datafeed-license \ --from-literal=licenseKey="${LICENSE_KEY}" # Create secret for the Honeycomb API key kubectl create secret generic datafeed-otlp \ --from-literal=api-key="${HONEYCOMB_API_KEY}" ``` Make sure to store these generated values securely for future maintenance and troubleshooting. Each value is: * `datafeed-postgres.password`: The password for your Postgres database. * `datafeed-ssh-salt.salt`: A random 32-char string used for hashing SSH public keys. * `datafeed-shepherd.workOSApiKey`: The WorkOS API key provided to you by Prequel. * `datafeed-shepherd.apiKey`: A random 32-char string used for admin API authentication. * `datafeed-shepherd.authToken`: A random 32-char string used to encrypt/decrypt authentication tokens. * `datafeed-license.licenseKey`: The license key provided to you by Prequel. * `datafeed-otlp.api-key`: The Honeycomb API key provided to you by Prequel. Fill in the `aws_on_prem_values_overrides.yaml` for the Prequel Helm chart. The following values should be set from the secrets created above: * `postgresDb.secretName`: `datafeed-postgres` or the name of the secret created for Postgres DB. * `postgresDb.passwordSecretKey`: `password` or the key in the secret created for Postgres DB that contains the password. * `sshSaltSecretName`: `datafeed-ssh-salt` or the name of the secret created for SSH salt. * `sshSaltSecretKey`: `salt` or the key in the secret created for SSH salt that contains the salt. * `shepherd.secretName`: `datafeed-shepherd` or the name of the secret created for Shepherd service. * `shepherd.workOS.apiKeySecretKey`: `workOSApiKey` or the key in the secret created for Shepherd service that contains the WorkOS API key provided to you by Prequel. * `shepherd.apiKeySecretKey`: `apiKey` or the key in the secret created for Shepherd service that contains the admin API key. * `shepherd.authTokenSecretKey`: `authToken` or the key in the secret created for Shepherd service that contains the authentication token key. * `licenseKeySecretName`: `datafeed-license` or the name of the secret created for the license key. * `licenseKeySecretKey`: `licenseKey` or the key in the secret created for the license key that contains the license key. Install the Prequel Helm chart. ```shell title="Install Helm chart" icon="terminal" theme={null} helm install prequel datafeed-{chart_version}.tgz -f aws_on_prem_values_overrides.yaml ``` The cluster should now be up and running. Nice work, we're almost there! Grab the address of the ingress / LB for the Prequel deployment. ```shell title="Get ingress address" icon="terminal" theme={null} kubectl get ing ``` and look for the `ADDRESS` field. In your domain settings, create DNS records for the three hosts used by Prequel. Specifically, for each, create a `CNAME` record which points to the address from the previous step. ```text title="DNS records" icon="file-lines" expandable theme={null} prequel.your-domain.com # the domain you'll use when hitting the API. prequel-admin.your-domain.com # the UI that admins on your team will use to manage Prequel. data-connect.your-domain.com # the domain your customers will use to connect their data warehouse ``` Notify your Prequel counterpart that the deployment is ready to roll. They'll guide you through next steps: configuring your first source. #### Updating Prequel We'll notify you when a new release is available, and provide you with the release tag. You can then run the following command to update your deployment to the new release. ```shell title="Upgrade Helm release" icon="terminal" theme={null} helm upgrade prequel datafeed-{chart_version}.tgz --reuse-values --set image.tag={provided_release_tag} ``` * Validate that you have access to the Terraform directory and Helm chart we sent over. If not, please email or Slack `support@prequel.co` to request access. * Create the dedicated cloud project where you'd like Prequel to run. We typically recommend creating a new cloud project for this, which allows all resources to be fully sandboxed from any other existing infrastructure. Take a look through `variables.tf` and fill in the required values. We have a `terraform.tfvars.example` that can be your reference. Perform a Terraform dry-run and double check that everything looks good. ```hcl title="Terraform" theme={null} terraform plan ``` Terraform the `main.tf` file. This will create all the necessary infrastructure for Prequel to run. Save the output variables, you'll need them later. ```hcl title="Terraform" theme={null} terraform apply ``` Update your DNS records to point to the `prequel-ingress-ip` returned by the Terraform script. You'll need to create three DNS records. ```text title="DNS records" icon="file-lines" expandable theme={null} prequel.your-domain.com # the domain you'll use when hitting the API. prequel-admin.your-domain.com # the UI that admins on your team will use to manage Prequel. data-connect.your-domain.com # the domain your customers will use to connect their data warehouse ``` **Important**: Workload Identity must be configured to allow Kubernetes service accounts to assume the GCP service account created by Terraform. This is required for Prequel services to access GCP resources. Verify that Workload Identity is enabled on your GKE cluster. If you used the provided Terraform configuration, this should already be enabled. You can check with: ```shell title="Verify Workload Identity" icon="terminal" expandable theme={null} # For zonal clusters gcloud container clusters describe {your_cluster_name} --zone={your_zone} --project={your_cluster_project_id} --format="value(workloadIdentityConfig.workloadPool)" # For regional clusters gcloud container clusters describe {your_cluster_name} --region={your_region} --project={your_cluster_project_id} --format="value(workloadIdentityConfig.workloadPool)" ``` The output should show `{your_cluster_project_id}.svc.id.goog`. If this is empty, Workload Identity is not enabled. **Note**: If your GKE cluster is in a different project than your service account, replace `{your_cluster_project_id}` with the actual project ID where your cluster is deployed. Create the IAM policy binding to allow the Kubernetes service account to impersonate the GCP service account: ```shell title="Bind service accounts" icon="terminal" expandable theme={null} # Replace the placeholder values with your actual values # {your_cluster_project_id} = Project ID where your GKE cluster is deployed # {your_service_account_project_id} = Project ID where your service account is created # {service_account_name} = Name of your service account (from Terraform) gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:{your_cluster_project_id}.svc.id.goog[default/datafeed]" \ {service_account_name}@{your_service_account_project_id}.iam.gserviceaccount.com # Also bind for the animalcontrol service account gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member "serviceAccount:{your_cluster_project_id}.svc.id.goog[default/animalcontrol]" \ {service_account_name}@{your_service_account_project_id}.iam.gserviceaccount.com ``` Verify the workload identity binding: ```shell title="Verify binding" icon="terminal" theme={null} # Test that the binding was created successfully gcloud iam service-accounts get-iam-policy {service_account_name}@{your_service_account_project_id}.iam.gserviceaccount.com ``` You should see the workload identity bindings in the output. Authenticate to the Kubernetes cluster created during infrastructure setup. Create the following Kubernetes secrets required by the Prequel deployment. ```shell title="Create Kubernetes secrets" icon="terminal" expandable theme={null} # Generate and store secure random values in environment variables export POSTGRES_PASSWORD={your_db_password} export WORKOS_API_KEY={workos_api_key} export SSH_SALT={your_generated_ssh_salt} export ADMIN_API_KEY={your_generated_admin_api_key} export AUTH_TOKEN_KEY={your_generated_auth_token_key} # Store the keys provided to you by Prequel export LICENSE_KEY={license_key_from_prequel} export HONEYCOMB_API_KEY={honeycomb_api_key_from_prequel} # Create secret for Postgres DB credentials kubectl create secret generic datafeed-postgres \ --from-literal=password="${POSTGRES_PASSWORD}" # Create secret for SSH salt (used for hashing public keys) kubectl create secret generic datafeed-ssh-salt \ --from-literal=salt="${SSH_SALT}" # Create secret for Shepherd service kubectl create secret generic datafeed-shepherd \ --from-literal=apiKey="${ADMIN_API_KEY}" \ --from-literal=authToken="${AUTH_TOKEN_KEY}" \ --from-literal=workOSApiKey="${WORKOS_API_KEY}" # Create secret for the Prequel license key kubectl create secret generic datafeed-license \ --from-literal=licenseKey="${LICENSE_KEY}" # Create secret for the Honeycomb API key kubectl create secret generic datafeed-otlp \ --from-literal=api-key="${HONEYCOMB_API_KEY}" ``` Make sure to store these generated values securely for future maintenance and troubleshooting. Each value is: * `datafeed-postgres.password`: The password for your Postgres database. * `datafeed-ssh-salt.salt`: A random 32-char string used for hashing SSH public keys. * `datafeed-shepherd.workOSApiKey`: The WorkOS API key provided to you by Prequel. * `datafeed-shepherd.apiKey`: A random 32-char string used for admin API authentication. * `datafeed-shepherd.authToken`: A random 32-char string used to encrypt/decrypt authentication tokens. * `datafeed-license.licenseKey`: The license key provided to you by Prequel. * `datafeed-otlp.api-key`: The Honeycomb API key provided to you by Prequel. Install the `cert-manager` Helm chart. ```shell title="Install cert-manager" icon="terminal" expandable theme={null} helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.18.2 \ --set crds.enabled=true \ --set ingressShim.defaultIssuerName=letsencrypt-prod \ --set ingressShim.defaultIssuerKind=ClusterIssuer \ --set ingressShim.defaultIssuerGroup=cert-manager.io ``` Fill `gcp_on_prem_values_overrides.yaml` based on your configurations. The following values should be set from the secrets created above: * `postgresDb.secretName`: `datafeed-postgres` or the name of the secret created for Postgres DB. * `postgresDb.passwordSecretKey`: `password` or the key in the secret created for Postgres DB that contains the password. * `sshSaltSecretName`: `datafeed-ssh-salt` or the name of the secret created for SSH salt. * `sshSaltSecretKey`: `salt` or the key in the secret created for SSH salt that contains the salt. * `shepherd.secretName`: `datafeed-shepherd` or the name of the secret created for Shepherd service. * `shepherd.workOS.apiKeySecretKey`: `workOSApiKey` or the key in the secret created for Shepherd service that contains the WorkOS API key provided to you by Prequel. * `shepherd.apiKeySecretKey`: `apiKey` or the key in the secret created for Shepherd service that contains the admin API key. * `shepherd.authTokenSecretKey`: `authToken` or the key in the secret created for Shepherd service that contains the authentication token key. * `licenseKeySecretName`: `datafeed-license` or the name of the secret created for the license key. * `licenseKeySecretKey`: `licenseKey` or the key in the secret created for the license key that contains the license key. Install the Prequel Helm chart. ```shell title="Install Helm chart" icon="terminal" theme={null} helm install prequel datafeed-{chart_version}.tgz -f gcp_on_prem_values_overrides.yaml ``` Notify your Prequel counterpart that the deployment is ready to roll. They'll guide you through next steps: configuring your first source. #### Updating Prequel We'll notify you when a new release is available, and provide you with the release tag. You can then run the following command to update your deployment to the new release. ```shell title="Upgrade Helm release" icon="terminal" theme={null} helm upgrade prequel datafeed-{chart_version}.tgz --reuse-values --set image.tag={provided_release_tag} ``` # AWS role assumption Source: https://docs.prequel.co/export/deployment/iam-role-assumption Understanding AWS role assumption and trust policies for cross-account and cross-cloud access. ## Understanding AWS role assumption In AWS, role assumption lets one role temporarily impersonate another by requesting short-lived credentials via AWS STS. It is a secure way to delegate access without passing around secret keys. In AWS, a Trust Policy on the target role defines which identities are allowed to assume it. ### Example AWS to AWS trust policy ```json title="AWS to AWS trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::123456789876:role/SomeDataTransferService" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "1234e567-89ab-cdef-0123-456789abcdef" } } } ] } ``` ### Example GCP to AWS trust policy ```json title="GCP to AWS trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["sts:AssumeRoleWithWebIdentity"], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "123456789012345678901", "accounts.google.com:oaud": "1234e567-89ab-cdef-0123-456789abcdef" } } } ] } ``` ## Interpreting the trust policy The Trust Policy that is attached to the target role will include 2 conditions. ### Condition 1: scoping the permission to the intended principal * For AWS to AWS this looks like: ```json title="AWS principal condition" icon="brackets-curly" expandable theme={null} /... "Principal": { "AWS": "arn:aws:iam::123456789876:role/SomeDataTransferService" } /... ``` * For GCP to AWS this looks like: ```json title="GCP principal condition" icon="brackets-curly" expandable theme={null} /... "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "123456789012345678901" } /... ``` This condition ensures that only the designated assuming role can act as the target role. This is regardless of the ExternalId. For example, in the case of connecting to an S3 bucket, this condition tells AWS "there's a specific service account that is allowed to assume my AWS role in order to write data into this bucket". ### Condition 2: preventing the confused deputy problem (ExternalId) * For AWS to AWS this looks like: ```json title="AWS ExternalId condition" icon="brackets-curly" expandable theme={null} "StringEquals": { "sts:ExternalId": "1234e567-89ab-cdef-0123-456789abcdef" } ``` * For GCP to AWS this looks like: ```json title="GCP audience condition" icon="brackets-curly" expandable theme={null} "StringEquals": { "accounts.google.com:oaud": "1234e567-89ab-cdef-0123-456789abcdef" } ``` This condition ensures that a bad actor (in this case, a different account holder) cannot establish a connection by supplying another account holder's S3 bucket & role ARN (that previously granted permission to their own account), and coerce the "data transfer service" to act on that account. When the transfer-service attempts to authenticate into the legitimate AWS/S3 role, the transfer-service will pass the Recipient ID of the bad actor, and AWS will block the authentication request, saying "The External ID provided is BadActor123 because it is attempting to act on behalf of Recipient BadActor123, but the Trust Policy on the S3 account is configured only for External ID LegitimateActor456". #### What is the ExternalId? The transfer service maintains an ExternalId (Recipient ID) for every account. These Recipient IDs are guaranteed to be unique, immutable, and are not configurable. # Observability for self-hosted Source: https://docs.prequel.co/export/deployment/logs-for-self-hosted Configuring logs and observability for your self-hosted deployment Structured logs that every Prequel service writes to std-out. Export OpenTelemetry traces and metrics to your observability stack. The platform events Prequel records, reviewable through the logs API. Subscribe to event types and push them to your systems as they happen. ## Std-out logs By default, all Prequel services surface structured logs to std-out. The structure of these logs should be somewhat intuitive, but please get in touch with the Prequel team if you have questions about the specific structure of certain logs. To facilitate debugging, we recommend ingesting these logs into AWS CloudWatch, GCP Logs Explorer, or your preferred logging solution. **Unsafe traces** On occasion, an external dependency not controlled by Prequel (such as a database driver) will produce a trace that might have the potential to contain transfer data. These traces are tagged with the `unsafe_trace` key in any Prequel structured log. To ensure compliance with your own user-data retention policies, we suggest either not ingesting those keys to logs, or ensuring that your logs are in line with any retention policies you may be bound to. See the relevant [GCP documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/exclude-logs#add-filter) and [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html) on how to avoid ingesting those logs. ## Observability Prequel code is instrumented with the Open Telemetry (OTEL) standard. Self-hosted deployments give the ability to export those telemetry events to the observability solution of your choice. We offer native support for solutions including Datadog, Elastic, Honeycomb, and more. Please get in touch with the team for information on how to set these up. # Prequel IPs Source: https://docs.prequel.co/export/deployment/prequel-ips How to find the Prequel static IPs The static IP you use during configuration will depend on your chosen deployment option. Based on your deployment, below are instructions for finding your IP. ## Prequel static IPs | Deployment | Static IPs | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Cloud Hosted (US) *(most popular)* | `35.192.85.117` | | Cloud Hosted (EU) | `104.199.49.149` | | Shared Cloud | Your Prequel deployment contact can share your static IPs, or you can find them in the shared GCP or AWS console. | | Cloud Prem | After configuring the static IPs during deployment, you can find your static IPs in the GCP or AWS console. | ## Why is this important? To enable secure cross-cloud reading and writing of data, the Prequel service is assigned a set of static IPs through which the service will connect to databases. The sources and destinations are configured to add read and write access (respectively) from these IP addresses only. # Azure Blob Storage Source: https://docs.prequel.co/export/destinations/abs Configuring your Azure Blob Storage destination. 1. In the Azure portal, navigate to the **Storage accounts** service and click **+ Create**. 2. In the "Basics" tab of the "Create a storage account" form, fill in the required details. 3. In the "Advanced" settings, under "Security" make sure **Enable storage account key access** is turned on. You may turn off (deselect) "Allow enabling public access on containers". Under "Data Lake Storage Gen2", select **Enable hierarchical namespace**. ![](https://storage.googleapis.com/prequel_docs/images/azure-settings-toggles.png "azure settings toggles.png") 4. In the "Networking" settings, you may limit "Network access" to either **Enable public access from all networks** or **Enable public access from selected virtual networks and IP addresses**. If the latter is selected, be sure to add the service's static IP to the address range of the chosen virtual network. All other settings can use the default selections. **Static IP** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. 5. In the "Data protection" settings, you must turn off **Enable soft delete for blobs**, **Enable soft delete for containers**, and **Enable soft delete for file shares**. ![](https://storage.googleapis.com/prequel_docs/images/azure-turn-off-settings.png "disable default soft delete settings.png") 6. Once the remaining options have been configured to your preference, click **Create**. 1. In the Azure portal, navigate to the **Storage accounts** service and click on the account that was created in the previous step. 2. In the navigation pane, under "Data storage", click **Containers**. Click **+ Container**, choose a name for the container, and click **Create**. **Recommendation: dedicated container for data transfers** Use a unique container for these transfers. This: * Prevents resource contention with other workloads * Avoids accidental data loss from mixed lifecycle or cleanup rules * Improves security by reducing surface area and enabling tighter, destination-scoped policies 3. In the navigation pane, under "Security + networking", click **Shared access signature**. 4. Update the required accessible services and permissions: 1. Under "Allowed services": select **Blob** and **File**. 2. Under "Allowed resource types": select **Container** and **Object**. 3. Under "Allowed permissions": select **Read**, **Write**, **List**, **Add**, and **Create** (optionally select **Delete** and **Permanently Delete** if you wish to let connection tests automatically clean up test files). 5. Select a "Start and expiry date/time" based on your security posture (e.g., set the expiration date 6 months into the future), and click **Generate SAS and connection string**. 6. Make a note of the **SAS token** that is generated. ![](https://storage.googleapis.com/prequel_docs/images/azure-setup-details.png "generate SAS token and connection string.png") Use the following details to complete the connection setup: **storage account name**, **container name**, your chosen **folder name** for the data, and your **Storage account SAS token**. ## Permissions checklist * SAS token includes: read, write, list, add, and create permissions (delete and permanently delete are optional for test connection cleanup) **Connection Test File Cleanup (Optional)** By default, a connection test is performed against the destination during initial configuration. This test writes temporary artifacts under the prefix `_test_connection/` (located at either the container root, or inside the custom folder prefix configured on the destination, if present). If you do not grant delete permissions (**Delete** and **Permanently Delete**), the connection test will still succeed, but these test files will remain in your container. To automatically clean them up and avoid clutter or minor storage charges, you can optionally configure an **Azure Lifecycle Management** rule on your storage account to delete blobs matching the prefix `_test_connection/` (or `/_test_connection/` if a custom folder was configured) after 1 day, or delete them manually. * Container exists in the intended account/region * If using network restrictions, the egress IP is allowed ## FAQ Data lands in Hive-style partitions per model: `//dt=/_.`. To write to the container root, enter `.` as the folder name. `Parquet` (default/recommended), `CSV`, and `JSON`/`JSONL`. Files are automatically split; multiple files may be written per model per transfer. Each transfer writes a manifest file per model under `_manifests`. Files are written per model per transfer in the format: `_manifests//dt=/manifest_{transfer_id}.json`. Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys prioritizing the most recent transfer window; manifest files can help bound the set of files to read. No explicit size/row limits for Blob Storage; files are split automatically based on volume and performance heuristics. # Athena Source: https://docs.prequel.co/export/destinations/athena Configuring your AWS Athena destination. ## Prerequisites * By default, Athena authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` ### Create Athena target bucket Follow these steps to create a bucket to be used for staging data before transferring to a destination. 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name**, select an **AWS Region**, and modify any of the default settings as desired. Note: **Object Ownership** can be set to "**ACLs disabled**" and **Block Public Access settings for this bucket** can be set to "**Block all public access**" as recommended by AWS. Make note of the Bucket name and AWS Region. 4. Click **Create bucket**. ### Create Athena access policy 1. Navigate to the **IAM** service page, click on the **Policies** navigation tab, and click **Create policy**. 2. Click the JSON tab, and paste the following policy, being sure to replace `ACCOUNT_ID`, `WORKGROUP`, `BUCKET_NAME` and `SCHEMA` with the your account information. * `WORKGROUP` should be `primary` unless otherwise specified during connection configuration. * `BUCKET` should refer to the bucket created in the previous step. * `SCHEMA` used below does not need to be created ahead of time. If it does not exist, it will be created automatically before transferring data. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAthenaAccess", "Effect": "Allow", "Action": [ "athena:GetQueryResults", "athena:StartQueryExecution", "athena:StopQueryExecution", "athena:StartSession", "athena:GetDatabase", "athena:GetDataCatalog", "athena:GetWorkGroup", "athena:GetTableMetadata", "athena:GetQueryExecution" ], "Resource": [ "arn:aws:athena:*:ACCOUNT_ID:workgroup/WORKGROUP" ] }, { "Sid": "AllowGlueAccessToDestinationDatabaseAndTables", "Effect": "Allow", "Action": [ "glue:GetDatabases", "glue:GetDatabase", "glue:GetTables", "glue:GetTable", "glue:GetPartitions", "glue:CreateTable", "glue:CreateDatabase", "glue:UpdateTable", "glue:DeleteTable" ], "Resource": [ "arn:aws:glue:*:ACCOUNT_ID:catalog", "arn:aws:glue:*:ACCOUNT_ID:database/SCHEMA", "arn:aws:glue:*:ACCOUNT_ID:database/default", "arn:aws:glue:*:ACCOUNT_ID:table/SCHEMA/*" ] }, { "Sid": "AllowS3AccessToBucket", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::BUCKET_NAME", "arn:aws:s3:::BUCKET_NAME/*" ] } ] } ``` **KMS encryption (optional)** If your S3 bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS policy statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. **Athena vs. S3 permissions** Because Athena uses S3 as the underlying storage layer, the Resource access requested in the policy is scoped down via resource-specific permissions in the S3 actions. 3. Click through to the **Review** step, choose a **name** for the policy, for example, `transfer-service-policy` (this will be referenced in the next step), add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy (from the prerequisite) to allow AssumeRole access to this role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Role based authentication is the preferred authentication mode for Athena based on AWS recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. 1. Navigate to the **IAM** service page. 2. Navigate to the **Users** navigation tab, and click **Add users**. 3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**. 4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**. 5. Click **Next: Review** and click **Create user**. 6. In the **Success** screen, record the **Access key ID** and the **Secret access key**. 1. Use the following details to complete the connection setup: **database**, **schema**, **workgroup**, **bucket name**, **bucket region**, and **IAM Role ARN**. Follow these guidelines to manage your new Athena tables effectively: ### Optimize Iceberg queries To optimize the performance of your Iceberg tables, use the `OPTIMIZE` command. This command reorganizes the data in a way that improves query efficiency. Execute the following query periodically: ```sql title="Optimize table" icon="database" theme={null} OPTIMIZE iceberg_table REWRITE DATA; ``` ### Set vacuum properties Iceberg tables can accumulate snapshots over time, which can affect performance. To manage this, set the maximum age for snapshots that the vacuum process should retain: ```sql title="Set vacuum properties" icon="database" theme={null} ALTER TABLE iceberg_table SET TBLPROPERTIES ( 'vacuum_max_snapshot_age_seconds'='259200'); ``` The default setting is `432000` seconds, we recommend only updating this if you notice degrading performance. ### Perform time travel queries Iceberg supports accessing historical data snapshots using time travel queries. This feature allows you to query the table as it appeared at a previous point in time, which is useful for audits and rollbacks: ```sql title="Time travel query" icon="database" theme={null} SELECT * FROM iceberg_table FOR TIMESTAMP AS OF timestamp; ``` Replace `timestamp` with the specific UNIX timestamp of the snapshot you wish to query. ## Permissions checklist * IAM role trust policy allows the service to assume the role. * IAM policy includes all Athena actions on the target workgroup ARN. * IAM policy includes all Glue actions on the target catalog, database, and tables. * IAM policy includes `s3:PutObject, s3:ListBucket, s3:GetBucketLocation, s3:GetObject, s3:DeleteObject` on the staging bucket and its contents. * If using KMS encryption: `kms:GenerateDataKey` and `kms:Decrypt` granted on the key. ## FAQ We use IAM role-based authentication. We assume the IAM role you configure using short-lived credentials. No static access keys are required. All access is scoped to the permissions defined in the role's IAM policy. Athena uses the AWS Glue Data Catalog to store and manage table metadata. The Glue permissions allow the service to create and update table definitions as data is synced. No. If the Glue database does not exist, it is created automatically before the first transfer. The `glue:CreateDatabase` permission in the policy enables this. If you prefer to use an existing database, remove `glue:CreateDatabase` from the policy and provide the existing database name. # Aurora MySQL Source: https://docs.prequel.co/export/destinations/aurora-mysql Configuring your Aurora MySQL destination. ## Prerequisites * If your MySQL database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Allow write access to a portion of your Aurora MySQL database. ### Configure the security group 1. In your **Amazon RDS** > **Databases** list, click the MySQL instance you want to send data to. 2. In the database page, in the **Connectivity & security** tab, make note of the **Endpoint** and the **Port** number. Note that you may need to select the "**Writer instance**" in the DB identifier list to reveal the endpoint. ![](https://storage.googleapis.com/prequel_docs/images/aws-mysql-endpoint-port.png "mysql endpoint port.png") 3. To ensure that the destination is accessible from outside your VPC, click "**Modify**" in the top right, and in the "**Connectivity**" section, within the **Additional configuration** dropdown, confirm the **Publicly accessible** setting is set to **Yes** . Note that it is still only accessible through whitelisted IPs at this point. If connecting via SSH tunnel, this can be set to **No**. ![](https://storage.googleapis.com/prequel_docs/images/aws-mysql-publicly-accessible.png "mysql publicly accessible.png") 4. Returning to the database page, within the "**Writer instance**" details, click one of the VPC security groups (usually `default`). Note: VPC groups are permissive (vs. restrictive) and for instances with multiple VPC security groups, only one needs to be configured with the new inbound rule. ![](https://storage.googleapis.com/prequel_docs/images/aws-mysql-default-security-group.png "vsg.png") 5. In the **Security Groups** section, select the **Inbound rules** tab. 6. Click **Edit inbound rules** and then click **Add rule**. 7. Edit the newly created rule of type **Custom TCP** with the **Port range** noted in the first step (usually `3306`) and a `Custom` **Source** value that includes all of the service IPs. Note: you will need to add `/32` to the end of each IP (CIDR notation). If connecting via SSH tunnel, use the bastion server's security group as the **Source** instead. 8. Click **Save rules**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-add-rule.png "add rule.png") ### Configure network ACLs (access control list) For database instances in a VCP 1. In your RDS dashboard, select the MySQL instance. 2. Click the link to the instance's VPC. 3. Click the **VPC ID**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-id.png "vpc id.png") 4. In the **Details** section, click on the link under **Main network ACL**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-main-network-acl-id.png "network acl id.png") 5. Click on the network ACL ID. ![](https://storage.googleapis.com/prequel_docs/images/postgres-network-acl-id.png "network acl id.png") #### Edit the inbound rules 6. Click on the **Inbound rules** tab, and check if there is an existing rule with a Source of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to **Edit outbound rules**.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-inbound-rules.png "inbound rules.png") 7. Create the inbound rule (if it doesn't exist). Click **Edit inbound rules** and either **Add new rule** or edit an existing rule to allow access to the **port number** of your database instance (usually `5432`) from the Prequel static IP. Click **Save changes**. #### Edit the outbound rules 8. In the ACL menu, select the **Outbound rules** tab, and check if there is an existing rule with a Destination of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to the next step.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-outbound-rules.png "outbound rules.png") 9. Create the outbound rule (if it doesn't exist). Click **Edit outbound rules** and edit the rules to allow outbound traffic to ports 1024-65535 for **Destination** `0.0.0.0/0`. ### Optional: SSH tunneling If your database is not publicly accessible, SSH tunneling through a bastion host is supported. 1. Allow inbound SSH (port `22`) from the static egress IP on the bastion host's security group. 2. On the bastion host, create an SSH user and add the service's public key to `~/.ssh/authorized_keys`. Contact support to obtain the public key. 3. Add the bastion host's security group to the database's inbound rules in place of the static egress IP (see step 7 above). 4. Supply the bastion host address, port, and username when configuring the destination. Create a database user to perform the writing of the source data. 1. Open a connection to your Aurora MySQL database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create user" icon="database" theme={null} CREATE USER @'%' IDENTIFIED BY ''; ``` 3. Grant user required privileges on the database. ```sql title="Grant privileges" icon="database" theme={null} GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW ON *.* TO @'%'; ``` **If the `schema/database` already exists:** By default, the service creates a new schema (*in MySQL, `schema` is synonymous with `database`*). If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using `GRANT ALL PRIVILEGES ON .* TO @'%';` Use the following details to complete the connection setup: **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password**. ## Permissions checklist * RDS security group inbound rules allow the service's egress IP on the configured port (usually `3306`). If using SSH tunneling, allow the egress IP on port 22 on the bastion host's security group instead. * Network ACL inbound rules allow traffic on the configured database port; outbound rules allow ports 1024-65535. * Database user has `SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW` on `*.*`. * If using a pre-created schema: user has `GRANT ALL PRIVILEGES ON .*`. ## FAQ We connect using the credentials you provide over TCP. Access is restricted by the RDS security group inbound rules, which should allow only the service's static egress IP on the configured port. SSH tunneling through a bastion host is supported for databases not publicly accessible. Aurora MySQL 3.x (MySQL 8.0-compatible) and above are fully supported. Aurora MySQL 2.x (MySQL 5.7-compatible) is best-effort only. No. The schema is created automatically on first sync. If you pre-create it, grant `ALL PRIVILEGES ON .*` to the writer user. # AWS RDS & AWS Aurora Source: https://docs.prequel.co/export/destinations/aurora-postgres Configuring your AWS Postgres (RDS or Aurora) destination. ## Prerequisites * If your Postgres database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Allow write access to a portion of your RDS or Aurora PostgreSQL database. ### Configure the security group 1. In your **Amazon RDS** > **Databases** list, click the PostgreSQL instance you want to send data to. 2. In the database page, in the **Connectivity & security** tab, make note of the **Endpoint** and the **Port** number. ![](https://storage.googleapis.com/prequel_docs/images/postgres-endpoint.png "endpoint + port.png") 3. In the **Security** section, ensure that the **Publicly accessible** setting is set to **Yes** to ensure that the destination is accessible from outside your VPC. Note that it is still only accessible through whitelisted IPs at this point. If connecting via SSH tunnel, this can be set to **No**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-publicly-accessible.png "publicly accessible.png") 4. Click one of the VPC security groups (usually `default`). Note: VPC groups are permissive (vs. restrictive) and for instances with multiple VPC security groups, only one needs to be configured with the new inbound rule. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-security-groups.png "vsg.png") 5. In the **Security Groups** section, select the **Inbound rules** tab. 6. Click **Edit inbound rules** and then click **Add rule**. 7. Edit the newly created rule of type **Custom TCP** with the **Port range** noted in the first step (usually `5432`) and a `Custom` **Source** value that includes all of the service IPs. Note: you will need to add `/32` to the end of each IP (CIDR notation). If connecting via SSH tunnel, use the bastion server's security group as the **Source** instead. 8. Click **Save rules**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-add-rule.png "add rule.png") ### Configure network ACLs (access control list) For database instances in a VCP 1. In your RDS dashboard, select the PostgreSQL instance. 2. Click the link to the instance's VPC. 3. Click the **VPC ID**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-id.png "vpc id.png") 4. In the **Details** section, click on the link under **Main network ACL**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-main-network-acl-id.png "network acl id.png") 5. Click on the network ACL ID. ![](https://storage.googleapis.com/prequel_docs/images/postgres-network-acl-id.png "network acl id.png") #### Edit the inbound rules 6. Click on the **Inbound rules** tab, and check if there is an existing rule with a Source of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to **Edit outbound rules**.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-inbound-rules.png "inbound rules.png") 7. Create the inbound rule (if it doesn't exist). Click **Edit inbound rules** and either **Add new rule** or edit an existing rule to allow access to the **port number** of your database instance (usually `5432`) from the Prequel static IP. Click **Save changes**. #### Edit the outbound rules 8. In the ACL menu, select the **Outbound rules** tab, and check if there is an existing rule with a Destination of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to the next step.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-outbound-rules.png "outbound rules.png") 9. Create the outbound rule (if it doesn't exist). Click **Edit outbound rules** and edit the rules to allow outbound traffic to ports 1024-65535 for **Destination** `0.0.0.0/0`. ### Optional: SSH tunneling If your database is not publicly accessible, SSH tunneling through a bastion host is supported. 1. Allow inbound SSH (port `22`) from the static egress IP on the bastion host's security group. 2. On the bastion host, create an SSH user and add the service's public key to `~/.ssh/authorized_keys`. Contact support to obtain the public key. 3. Add the bastion host's security group to the database's inbound rules in place of the static egress IP (see step 7 above). 4. Supply the bastion host address, port, and username when configuring the destination. Create a database user to perform the writing of the source data. 1. Open a connection to your Amazon RDS PostgreSQL database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD ''; ``` 3. Grant user `create` and `temporary` privileges on the database. `create` allows the service to create new schemas and `temporary` allows the service to create temporary tables. ```sql title="Grant privileges" icon="database" theme={null} GRANT CREATE, TEMPORARY ON DATABASE TO ; ``` **If the `schema` already exists:** By default, the service creates a new schema based on the destination configuration (in the next step). If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using `GRANT ALL ON schema TO ;` Use the following details to complete the connection setup: **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password**. ## Permissions checklist * RDS security group inbound rules allow the service's egress IP on the configured port (usually `5432`). If using SSH tunneling, allow the egress IP on port 22 on the bastion host's security group instead. * Network ACL inbound rules allow traffic on the configured database port; outbound rules allow ports 1024-65535. * Database user has `CREATE` and `TEMPORARY` on the target database. * If using a pre-created schema: user has `GRANT ALL ON SCHEMA `. ## FAQ We connect using the credentials you provide over TCP. Access is restricted by the RDS security group inbound rules, which should allow only the service's static egress IP on the configured port. SSH tunneling through a bastion host is supported for databases not publicly accessible. Aurora PostgreSQL versions compatible with PostgreSQL 13 and above are fully supported. PostgreSQL 12-compatible versions are best-effort only. No. The schema provided in the destination configuration is created automatically on first sync. If you pre-create it, grant `ALL` on the schema to the writer user and you may remove the database-level `CREATE` permission (retain `TEMPORARY`). # Configuring AWS to GCP role assumption Source: https://docs.prequel.co/export/destinations/aws-to-gcp-role-assumption Configure a GCP Workload Identity Pool so an AWS role can impersonate your GCP service account. ## Prerequisites * You will need the name of the GCP service account that you previously created as part of your destination configuration. The name should look like `name@project.iam.gserviceaccount.com`. * You will need the provided AWS ARN and AWS Account ID of the role that will impersonate your GCP service account. * The ARN value should look like: `arn:aws:iam::123456789012:role/SomeRoleName`. * The AWS Account ID is the 12 digit number from the ARN value, e.g., `123456789012`. 1. In the GCP console, navigate to the **IAM & Admin** menu, click into the **IAM** tab, and find the GCP service account that you previously created as part of the Prerequisites. Click **Edit principal**. 2. In the menu that appears, click **+ Add Another Role**. Search for and add the "Workload Identity User" role to the service account. ![](https://storage.googleapis.com/prequel_docs/images/gcp_pool_workload_identity_user.png) 1. In the GCP console, navigate to **IAM & Admin** menu, click into the **Workload Identity Federation** tab, and click **Create Pool**. ![](https://storage.googleapis.com/prequel_docs/images/gcp_pool_create_pool.png) 2. Complete the details in the first step. For example, you may want to use a name like "transfer-service-pool" and a pool ID called "transfer-pool". Click **Continue**. 3. Complete the details in the second step. 1. For **Select Provider**, select **AWS** in the dropdown. 2. For **Provider Name**, enter a descriptive name of your data source. 3. For **AWS Account ID**, enter the Account ID (from the **Prerequisite**). ![](https://storage.googleapis.com/prequel_docs/images/gcp_pool_add_provider_to_pool.png) 4. Complete the third step, leave the provider attributes mapping as is by default. Finish creating the pool. 5. In the overview of the created Workload Identity Pool, click **Grant Access**. ![](https://storage.googleapis.com/prequel_docs/images/gcp_pool_grant_access.png) 6. In the menu that appears, complete the setup flow: 1. Select **Grant access using Service Account impersonation**. 2. In the dropdown, select the Service Account you created (from the **Prerequisite**). 3. Enter the **Service principals (identities that can access the Service Account)**: 1. For the **attribute name**, select `aws_role`. 2. For the **attribute value**, enter the AWS ARN (from the **Prerequisite**). 4. Click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp_pool_grant_access_to_service_account.png) 7. After saving, you will be able to download a JSON file of metadata. This is the metadata you will need to complete your connection. ![](https://storage.googleapis.com/prequel_docs/images/gcp_pool_download_metadata.png) # Azure SSH tunneling Source: https://docs.prequel.co/export/destinations/azure-ssh-tunneling Configure an Azure bastion server to accept an SSH connection from our service. Follow these steps to configure an Azure bastion server to accept an SSH connection from our service. ## Prerequisites * A public SSH key to add to the bastion machine to authenticate the SSH connection. *Ask your contact at our service if they have not already provided one to you.* ### Selecting your bastion server If using SSH tunneling to connect to a destination database, we recommend using a standalone bastion server. This step will walk through provisioning the standalone bastion server. If using an existing server, skip to the next section about configuring network access. 1. In the Azure portal, search for "Virtual machines", and click into the **Virtual Machine** service. Click **Create** and then **Azure virtual machine**. 2. Select an image. We recommend a default Ubuntu image, such as the *Ubuntu Server 20.04 LTS - x64 Gen2* (the default option). 3. Select a type. We recommend a basic & affordable option such as the **Standard\_B1s**. *Because this bastion server will be used exclusively for SSH tunneling, memory, compute, and storage are not particularly important. The Standard\_B1s default configuration comes with sufficient Memory (1 GiB), vCPUs (2) and Storage (8 GB SSD) for about \$7/mo depending on region and market rates.* ### Configure network access 3. Click the **Networking** tab. Select a **Virtual network** to associate with the machine. *By default, you may be prompted to create a new virtual network, you may suggest the default suggestion.* 4. In the **Subnet** field, select the public subnet you want to associate with the virtual machine. *By default, you may be prompted to create a new virtual network, you may accept the default suggestion.* 5. In the **Public IP** dropdown, verify that a valid selection is selected (*do not choose None*). *By default, you may be prompted to create a new Public IP, you may accept the default suggestion.* 6. For NIC network security group, select **Advanced**. This will display the **Configure network security group** dropdown. 7. Click the **Create new** link under the **Configure network security group** dropdown to reveal the Create network security group panel. In the panel, click the **Add an inbound rule** link in the Inbound rules section. This will open the Add inbound security rule panel: 8. Fill in the following fields, leaving any unlisted fields as the default: 1. **Source:** Select the **IP Addresses** option. 2. **Source IP addresses/CIDR ranges:** Add our static IP. Optionally, add a description. For the correct IP, refer to the [IPs](/export/deployment/prequel-ips) section of the docs. \_ **Note**: you will not be able to SSH into this machine to configure the SSH user unless you also whitelist your machine's IP in the inbound rules. Consider temporarily adding that IP to facilitate the next step.\_ 3. **Source port ranges:** Enter **22**. 4. **Protocol:** Select **TCP**. 5. **Action:** Select **Allow**. 6. **Name:** Enter a descriptive name, for example: `ssh-tunnel-inbound`. 7. When finished, click **Add** to create the inbound rule. 9. Click "OK" and then **Review + create**. Review the details and click **Create** to create the tunnel. ### Make a note of your public and private IP addresses 1. In the Azure portal, search for "Virtual machines", and click into the **Virtual Machine** service. Click the name of the Virtual machine you just created. 2. In the **Networking** section of the **Properties** tab, make a note of the **Public IP address** and **Private IP address**. ### Create the SSH user Login to the bastion server and complete the steps below. 1. Create group ``: ```shell title="Create group" icon="terminal" theme={null} sudo groupadd ``` 2. Create user ``: ```shell title="Create user" icon="terminal" theme={null} sudo useradd -m -g ``` 3. Switch to the SSH user: ```shell title="Switch to SSH user" icon="terminal" theme={null} sudo su - ``` 4. Create the `.ssh` directory: ```shell title="Create .ssh directory" icon="terminal" theme={null} mkdir ~/.ssh ``` 5. Set permissions: ```shell title="Set directory permissions" icon="terminal" theme={null} chmod 700 ~/.ssh ``` 6. Navigate to the `.ssh` directory: ```shell title="Navigate to .ssh directory" icon="terminal" theme={null} cd ~/.ssh ``` 7. Create the `authorized_keys` file: ```shell title="Create authorized_keys file" icon="terminal" theme={null} touch authorized_keys ``` 8. Set permissions: ```shell title="Set file permissions" icon="terminal" theme={null} chmod 600 authorized_keys ``` 9. Add the public key to the `authorized_keys` file. *The key, including the "ssh-key" and comment, should be all on one line in the file, without linebreaks.* ```shell title="Add public key" icon="terminal" theme={null} echo "" > authorized_keys ``` 1. Find the database you'd like to connect to in your cloud console. 2. Find the **Connection security** section, and add a new rule. 3. If using an **Azure database**, add a new rule with the **Private IP** noted in Step 1 as the **Start IP** and **End IP**. Choose a name and click **Save**. # BigQuery Source: https://docs.prequel.co/export/destinations/bigquery Configuring your BigQuery destination. ## Prerequisites * By default, BigQuery authentication uses role-based access. You will need our service account name available to grant access. It should look like `some-name@some-project.iam.gserviceaccount.com`. ## Understanding role-based authentication in BigQuery **Two service accounts involved** * **Destination service account (in your GCP project):** You create this service account in Step 1. It has permissions to BigQuery and Cloud Storage and is the identity that performs work inside your project. * **Our service account:** Provided to you in the prerequisites. It does not have direct permissions to BigQuery or Cloud Storage. Instead, it is granted permission to "assume" the other service account role (using short-lived tokens via Service Account Token Creator/User), enabling least-privilege, auditable access without sharing keys. **See the end of the configuration guide for Frequently Asked Questions with the BigQuery destination** 1. In the GCP console, navigate to the **IAM & Admin** menu, click into the **Service Accounts** tab, and click **Create service account** at the top of the menu. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-service-account-menu.png "create service account menu.png") 2. In the first step, name the new **Destination service account** and click **Create and Continue**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-service-account-name-options.png "service account name options.png") 3. In the second step, grant the new **Destination service account** the **BigQuery User** role. This allows creating datasets, submitting load/query jobs, and accessing required metadata during setup. ![](https://storage.googleapis.com/prequel_docs/images/gcp-bigquery-user.png) **Alternative: Dataset Already Exists. Why:** use least-privilege when your dataset is pre-provisioned. * Project: grant `bigquery.jobs.create` to the **Destination service account**. * Dataset: grant **BigQuery Data Owner** OR a custom role including at minimum: `bigquery.datasets.get`, `bigquery.tables.create`, `bigquery.tables.delete`, `bigquery.tables.get`, `bigquery.tables.getData`, `bigquery.tables.list`, `bigquery.tables.update`, `bigquery.tables.updateData`, `bigquery.routines.get`, `bigquery.routines.list`. 4. Click **Done** to finish creating the account. 5. In the service accounts list, click the newly created **Destination service account** to open its details and make a note of the **email** (this is different from **our service account** from the prerequisites). 6. Grant access to the **Destination service account** using one of the following authentication methods: Navigate to the **Principals with access** tab, click **Grant Access**, and add the following principal and roles: * **Principal:** the **service account we provide** (see prerequisites) * **Roles to grant:** **Service Account Token Creator**, **Service Account User** **Why:** only when policy requires, not recommended and presents higher security risk than impersonation. * Generate a JSON key for the **Destination service account** and use it to authenticate. * Steps: IAM & Admin → Service Accounts → open the **Destination service account** → Actions → Manage keys → Add key → Create new key → Key type: JSON → Create. Securely store the key. ![](https://storage.googleapis.com/prequel_docs/images/gcp-grant-role.png) **Why:** authenticate with OAuth client credentials issued by your own identity provider. We fetch a token from your identity provider and exchange it with Google using Workload Identity Federation, so no long-lived GCP keys are shared and access is revocable at your identity provider. 1. Configure an OAuth client (client ID and client secret) in your identity provider. The token endpoint must be reachable over https from the public internet, support the `client_credentials` grant with `client_secret_basic` authentication, and issue **JWT** access tokens (opaque tokens are rejected by Google). 2. In your GCP project, create a **Workload Identity Federation** pool with an **OIDC provider** pointing at your identity provider's issuer. Set the provider's **allowed audiences** to the value your tokens carry in their `aud` claim. 3. Grant the **Destination service account** the **Workload Identity User** role for the federated principal matching your tokens' `sub` claim: `principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools//subject/`. 4. When creating the destination, choose the `client_secret_workload_identity_federation` auth method and provide the **client ID**, **client secret**, **token endpoint URL**, an optional **scope** (one or more unique space-delimited, case-sensitive scope values, e.g. `read write`), the **Destination service account email**, and the provider's full resource name as the **workload identity federation audience**: `//iam.googleapis.com/projects//locations/global/workloadIdentityPools//providers/`. 1. Log into the Google Cloud Console and navigate to **Cloud Storage**. Click **Create** to create a new bucket. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-gcs-bucket.png) 2. Choose a **name** for the bucket. Click **Continue**. Select a **location** for the staging bucket. Make a note of both the **name** and the **location** (region). **Choosing a `location` (region)** The location you choose for your staging bucket must match the location of your destination dataset in BigQuery. When creating your bucket, be sure to choose a region in which BigQuery is supported [(see BigQuery regions)](https://cloud.google.com/bigquery/docs/locations) * If the dataset **does not** exist yet, the dataset will be created for you in the same region where you created your bucket. * If the dataset **does** exist, the dataset region must match the location you choose for your bucket. 3. Click **Continue** and select the following options according to your preferences. Once the options have been filled out, click **Create**. 4. Ensure the bucket is **not public**. We recommend enabling **Uniform bucket-level access** and keeping all **Public access** blocked. 5. On the **Bucket details** page that appears, click the **Permissions** tab, and then click **Add**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-add-permission-to-bucket.png) 6. In the **New principals** field, add the **Destination service account** created in **Step 1**, select the **Storage Admin** role, and click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-storage-admin.png) **Alternative: Understanding GCS Bucket Scope. Why:** reduce privileges while preserving required functionality. * We strongly recommend using a new, **dedicated bucket dedicated solely to data transfers for data isolation and to simplify permissions management**. However, if policy requires tighter scope than Storage Admin, you can grant only the following minimum actions to the **Destination service account**: `storage.buckets.get`, `storage.objects.list`, `storage.objects.get`, `storage.objects.create`, `storage.objects.delete`. * How: use a custom role, or provide both **Storage Legacy Bucket Reader** + **Storage Object User**. **Optional: Add a short retention lifecycle policy** You may configure a lifecycle rule on the staging bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. In the bucket **Lifecycle** tab, add a rule with action "Delete object" and condition "Age: 2 days". Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. 1. Log into the Google Cloud Console and select the projects list dropdown. 2. Make note of the BigQuery **Project ID**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-project-id.png "project id.png") **Domain-restricted sharing supported** This connection supports Google Cloud organization policies that restrict identities by domain. If your organization enforces domain-restricted sharing, you can whitelist our principal according to Google's guidance on restricting identities by domain. See the Google Cloud documentation: [Restricting identities by domain](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains). Contact the team to receive the customer ID to add to your allow list. Use the following details to complete the connection setup: **Project ID**, **Bucket Name**, **Bucket Location**, **Destination Dataset Name**, and **Destination service account** name. ## Permissions checklist * Destination service account exists in your project. * Project: Destination service account has BigQuery User. If dataset is pre-created, instead grant project `bigquery.jobs.create` + dataset-level Data Owner (or custom role with the minimum dataset, table, and routine permissions listed above). * On the Destination service account: grant our service account the Service Account Token Creator and Service Account User roles. * Staging bucket is non-public and in the same region as the BigQuery dataset. * Staging bucket: Destination service account has Storage Admin. If using tighter scope, ensure minimal object and bucket permissions are granted. * Optional: lifecycle rule deletes objects after \~2 days. ## FAQ We use staging-assisted load to use BigQuery's native bulk-upload path, maximizing throughput to your destination. Data is not persisted in the staging bucket and is deleted after each transfer. You may optionally add a lifecycle rule to auto-delete objects after \~2 days. Yes. BigQuery is supported across all [GCP-supported regions](https://cloud.google.com/storage/docs/locations). Ensure your BigQuery dataset and staging bucket are located in the same region. GCP IAM services can often take up to 10 minutes to propagate. Please wait a few minutes and try again. You create one service account in your project with BigQuery/Storage permissions, and we use our service account to impersonate yours. This means we never handle your private keys, all operations appear in your audit logs, access is via short-lived tokens, and you can revoke access anytime through your own IAM permissions. Direct service account access is not supported. # ClickHouse Source: https://docs.prequel.co/export/destinations/clickhouse Configuring your ClickHouse destination. ## Prerequisites * If your ClickHouse security posture requires IP whitelisting, have our static IP available during the following steps. It will be required in Step 1. **SSH Tunneling Not Supported** SSH Tunneling is currently unsupported for Clickhouse destinations. Please ensure your Clickhouse destination is accessible over the public internet. Create a rule in a security group or firewall settings to whitelist: 1. incoming connections to your host and port (usually `9440`) from the static IP. 2. outgoing connections from ports `1024` to `65535` to the static IP. **Network allowlisting** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. Create a database user to perform the writing of the data. 1. Open a connection to your ClickHouse database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create user" icon="database" theme={null} CREATE USER @'%' IDENTIFIED BY ''; ``` **Password Rules** Passwords may only include alphanumeric characters (A-Z, a-z, 0-9), dashes (-), and underscores (\_). 3. Grant user required privileges on the database. ```sql title="Grant privileges" icon="database" expandable theme={null} GRANT SELECT ON information_schema.columns TO ; GRANT CREATE, INSERT, DROP, ALTER, OPTIMIZE, SHOW, TRUNCATE ON .* TO @'%'; grant CREATE TEMPORARY TABLE, S3 on *.* to @'%'; ``` **Understanding the `CREATE TEMPORARY TABLE, S3` permissions** The `CREATE TEMPORARY TABLE` and `S3` permissions are required to efficiently transfer data to ClickHouse. Under the hood, these permissions are used to stage data in object storage as compressed files, COPY INTO temporary tables, and finally merge into the target tables. By definition, the temporary table will not exist outside of the session. ClickHouse sources require a staging bucket to efficiently transfer data. Configure your staging bucket using one of the following types of ClickHouse supported object storage: * [S3 staging bucket](/export/destinations/staging-s3) * [GCS staging bucket](/export/destinations/staging-gcs) * Implicit **Using the `implicit` bucket option** ClickHouse supports the ability to configure staging resources with [environment credentials](https://clickhouse.com/docs/en/integrations/s3#managing-credentials). If this setting is enabled on your ClickHouse cluster, you may choose to use the configured implicit staging resources using the `implicit` option for the staging bucket selection. **Optional: Add a short retention lifecycle policy** If using S3 or GCS for staging, you may configure a lifecycle rule on the bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ### Granting ClickHouse Cloud role-based access to S3 If your ClickHouse instance runs on ClickHouse Cloud, you can have it authenticate to your S3 staging bucket using the same IAM role instead of access keys to avoid relying on long-lived static credentials. You may follow the same steps in the [S3 staging bucket configuration](/export/destinations/staging-s3) above, but you will need to add an additional trust policy statement to allow ClickHouse to assume the role too. ```json title="Trust policy statement" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { // Existing statements }, { "Effect": "Allow", "Principal": {"AWS": ""}, "Action": "sts:AssumeRole" } ] } ``` Replace `` with your ClickHouse instance's IAM ARN. To obtain the ARN, go to your ClickHouse Cloud account, navigate to **Settings** → **Network security information** → **View service details** and copy the **Service role ID (IAM)**. See the [ClickHouse Secure S3 documentation](https://clickhouse.com/docs/cloud/data-sources/secure-s3) for full details, including an automated CloudFormation setup option. **Connection Protocol** Use the ClickHouse TCP native protocol, not HTTPS. This is commonly exposed on port 9000. Use the following details to complete the connection setup: **host name**, **port**, **cluster**, **database name**, **schema name**, **username**, **password**, and staging bucket details. **Understanding the `database` vs. `schema` fields (`connection database` vs. `write database`)** Depending on the version of your integration, you may be asked for both a `database` and `schema`, or a `connection database` and `write database`. * `database` (also referred to as `connection_database`): is the **database** used to establish the connection with ClickHouse. * `schema` (also referred to as `write_database`): is the **database/schema** within which data will be written These can be (and often are) the same values, but do not need to be. ## Using the ClickHouse data **Querying ClickHouse data without duplicates** The resulting ClickHouse tables use the [ReplacingMergeTree](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replacingmergetree) table engine in order to efficiently upsert changes. To properly query this data, the `FINAL` keyword must be used when selecting from these tables guarantee duplicates are removed. For example: ```sql title="Query without duplicates" icon="database" expandable theme={null} SELECT * FROM schema.table FINAL WHERE foo = bar ORDER BY foo LIMIT 10; ``` ## Permissions checklist * User has `SELECT` on `information_schema.columns`. * User has `CREATE, INSERT, DROP, ALTER, OPTIMIZE, SHOW, TRUNCATE` on `.*`. * User has `CREATE TEMPORARY TABLE, S3` on `*.*`. * Staging bucket configured (S3, GCS, or implicit). * Firewall or security group allows the service's egress IP on the ClickHouse native protocol port (default: `9440` for TLS, `9000` for TCP). * Password uses only alphanumeric characters, dashes (`-`), or underscores (`_`). ## FAQ We connect using the username and password you configure over the ClickHouse native TCP protocol. Network access can be restricted by allowlisting the service's static egress IP in your firewall or security group. Note: SSH tunneling is not supported for ClickHouse destinations. # Databricks Source: https://docs.prequel.co/export/destinations/databricks Configuring your Databricks destination. ## Prerequisites * By default, this Databricks integration makes use of Unity Catalog data governance features. You will need Unity Catalog enabled on your Databricks Workspace. Create a new SQL warehouse for data writing. 1. Log in to the Databricks account. 2. In the navigation pane, click **SQL Warehouses**. 3. In the SQL Warehouses console, click **Create SQL Warehouse**. 4. In the **New SQL Warehouse** menu, choose a **Name** and configure the options for the new SQL warehouse. Under "Advanced options" ensure "Unity Catalog" is in the **On** position and click **Create**. ![](https://storage.googleapis.com/prequel_docs/images/databricks-new-sql-warehouse-ui-refresh.png "databricks-new-sql-warehouse-ui-refresh.png") 1. In your Databricks workspace, click your username or icon in the top right, click **Settings**, **Identity and access**, and next to the **Service Principals** options, click **Manage**. 2. Click the **Add service principal** button, click **Add new** in the modal, enter a display name and click **Add**. 3. Click on the newly created Service Principal, and under the **Secrets** tab, click **Generate secret**. 4. Enter a lifetime for the secret (e.g., 90 days, 180 days, or 365 days), click **Generate** and make a note of the **Secret** value and **Client ID**. 5. Navigate back to the **SQL Warehouses** section of your Workspace, click the **SQL Warehouses** tab, and select the **SQL Warehouse** you created in **Step 1**. Click **Permissions** in the top right, search for and select the **Service Principal** you created, select the **Can use** permission, and click **Add**. 6. In the Databricks UI, select the **Catalog** tab, and select the target **Catalog**. Within the catalog **Permissions** tab, click **Grant**. In the following modal, select the **Service principal** for which you generated the OAuth token, select `USE CATALOG`, and click **Grant**. 7. Under the target **Catalog**, select the target **schema** (e.g., `main.default`, or create a new target schema). Within the schema **Permissions** tab, click **Grant**. In the following modal, select the **principal** for which you generated the access token, and select either `ALL PRIVILEGES` or the following 9 privileges and then click **Grant**: * `USE SCHEMA` * `APPLY TAG` * `MODIFY` * `READ VOLUME` * `SELECT` * `WRITE VOLUME` * `CREATE MATERIALIZED VIEW` * `CREATE TABLE` * `CREATE VOLUME` Collect connection information and create an access token for the data transfer service. 1. In the **SQL Warehouses** console, select the SQL warehouse you created in **Step 1**. 2. Click the **Connection Details** tab, and make a note of the **Server hostname**, **Port**, and **HTTP path**. ![](https://storage.googleapis.com/prequel_docs/images/databricks-server-path-ui-refresh.png "databricks-server-path-ui-refresh.png") 3. Click the link to Create a **personal access token**. ![](https://storage.googleapis.com/prequel_docs/images/databricks-create-personal-access-token-ui-refresh.png "create_a_personal_access_token.png") 4. Click **Generate New Token**. 5. Name the token with a descriptive comment and assign the token lifetime. A longer lifetime will ensure you do not have to update the token as often. If PAT scopes are enabled on your workspace, select the `sql` scope. Click **Generate**. ![](https://storage.googleapis.com/prequel_docs/images/databricks-pat-scopes.png "databricks-pat-scopes.png") 6. In the pop up that follows, **copy the token** and securely save the token. **Using a Service Principal & a Personal Access Token ("PAT") tied to the Service Account** You may prefer to create a **Service Principal** with a PAT to use for authentication instead of using your account's Personal Access Token. To do so, use the following steps to create a Service Principal and generate a PAT on behalf of the Service Principal. 1. In your Databricks workspace, click your username or icon in the top right, click **Settings**, **Identity and access**, and next to the **Service Principals** options, click **Manage**. 2. Click the **Add service principal** button, click **Add new** in the modal, enter a display name and click **Add**. 3. Click on the newly created Service Principal, and under **Entitlements** select **Databricks SQL Access** and **Workspace Access**. Click **Update**, and make a note of the **Application ID** of your newly created Service Principal. 4. Back in the **Admin Settings** menu, click the **Advanced** section (under the **Workspace admin** menu). In the **Access Control** section, next to the **Personal Access Tokens** row, click **Permission Settings**. Search for and select the **Service Principal** you created, select the **Can use** permission, click **Add**, and then **Save**. 5. Navigate back to the **SQL Warehouses** section of your Workspace, click the **SQL Warehouses** tab, and select the **SQL Warehouse** you created in **Step 1**. Click **Permissions** in the top right, search for and select the **Service Principal** you created, select the **Can use** permission, and click **Add**. 6. Use your terminal to generate a **Service Principal Access Token** using your Personal Access Token generated above. Record the **token value**. This token can now be used as the access token for the connection. ```bash title="Generate access token" icon="terminal" expandable theme={null} curl --request POST "https://.cloud.databricks.com/api/2.0/token-management/on-behalf-of/tokens" \ --header "Authorization: Bearer " \ --data '{ "application_id": "", "lifetime_seconds": , "comment": "" }' ``` 7. In the Databricks UI, select the **Catalog** tab, and select the target **Catalog**. Within the catalog **Permissions** tab, click **Grant**. In the following modal, select the **principal** for which you generated the access token, select `USE CATALOG`, and click **Grant**. 8. Under the target **Catalog**, select the target **schema** (e.g., `main.default`, or create a new target schema). Within the schema **Permissions** tab, click **Grant**. In the following modal, select the **principal** for which you generated the access token, and select either `ALL PRIVILEGES` or the following 9 privileges and then click **Grant**: * `USE SCHEMA` * `APPLY TAG` * `MODIFY` * `READ VOLUME` * `SELECT` * `WRITE VOLUME` * `CREATE MATERIALIZED VIEW` * `CREATE TABLE` * `CREATE VOLUME` **IP Access Lists (optional)** If your workspace enforces Databricks IP Access Lists, allowlist the static egress IP(s) used by the data transfer service so connections and test sessions can open successfully. See Databricks documentation: [Databricks IP Access Lists](https://learn.microsoft.com/en-us/azure/databricks/security/network/front-end/ip-access-list). Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. Use the following details to complete the connection setup: **server hostname**, **HTTP path**, **catalog**, your chosen **schema name**, and authentication details. ## Permissions checklist * Workspace: Service principal or user has permission to use the target SQL warehouse ("Can use"). * Unity Catalog: `USE CATALOG` on the target catalog; appropriate privileges on the target schema (e.g., `USE SCHEMA`, `CREATE TABLE`, and read/write volume privileges as needed), or `ALL PRIVILEGES` on the schema. * If using Hive Metastore (instead of UC): select `hive` metastore explicitly and configure required object storage staging (bucket and credentials), with write/delete permissions. * Network: If IP Access Lists are enabled, the static egress IP(s) are allowlisted. **Optional: Add a short retention lifecycle policy (Hive Metastore only)** If using an S3 staging bucket with Hive Metastore, you may configure a lifecycle rule on the bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. In the bucket **Management** tab, click **Create lifecycle rule**, set an expiration action for current versions of objects with a 2-day age. Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ## FAQ Recommended: OAuth with a service principal. OAuth issues short-lived tokens, scopes access via the principal's entitlements and UC grants, and supports centralized rotation and revocation. Personal Access Tokens (PATs) are supported where policy requires, but they are long-lived bearer tokens and typically inherit broader, user-level permissions. The connection identity needs `Can use` on the SQL warehouse, `USE CATALOG` on the target catalog, and schema-level privileges to create/manage tables (or `ALL PRIVILEGES` on the schema). Missing `USE CATALOG` is a frequent cause of test-connection failures. Provide Server hostname, HTTP path, Catalog, Schema, and an OAuth client (service principal) or PAT with warehouse access. Collect host/path from the SQL Warehouses console. Yes. Unity Catalog is the default. For Hive Metastore, explicitly select `hive` in configuration and supply an S3 staging bucket and credentials. Using the wrong metastore type can cause discovery or permission issues. # Delta Lake Source: https://docs.prequel.co/export/destinations/delta-lake Configuring your Delta Lake destination. **Beta destination over 1 billion rows per month** Delta Lake is considered a beta destination for volumes exceeding 1 billion rows per month. Please contact the team if you are interested in using this destination for larger volumes. ## Setting up AWS S3 Delta Lake ### Prerequisites * By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` ### Create bucket 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**. 4. Click **Create bucket**. ### Create policy 1. Navigate to the **IAM** service page. 2. Navigate to the **Policies** navigation tab, and click **Create policy**. 3. Click the **JSON** tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen in Step 1. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` **KMS encryption (optional)** If your S3 bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS policy statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 4. Click **Next: Tags**, click **Next: Review**. 5. Name the policy, add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Role based authentication is the preferred authentication mode for S3 based on AWS recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. 1. Navigate to the **IAM** service page. 2. Navigate to the **Users** navigation tab, and click **Add users**. 3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**. 4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**. 5. Click **Next: Review** and click **Create user**. 6. In the **Success** screen, record the **Access key ID** and the **Secret access key**. Use the following details to complete the connection setup: **bucket name**, **bucket region**, and **role ARN**. ## Setting up Google Cloud Delta Lake ### Prerequisites * By default, GCS authentication uses role-based access. You will need our service account name available to grant access. It should look like `some-name@some-project.iam.gserviceaccount.com`. 1. In the GCP console, navigate to the **IAM & Admin** menu, click into the **Service Accounts** tab, and click **Create service account** at the top of the menu. 2. In the first step, name the service account that will be used to transfer data into Cloud Storage and click **Create and Continue**. Click **Continue** in the following optional step without assigning any roles. 3. In the **Grant users access to this service account** step, within the **Service account users role** field, enter the provided **Service account** (see prerequisite) and click **Done**. 4. Once successfully created, search for the created service account in the service accounts list, click the **Service account** name to view the details, and make a note of the **email** (note: this is a different email than the service's service account). 5. Grant access using one of the following authentication methods: Select the permissions tab, find the provided principal name (**Service account** from the prerequisite), click the **Edit principal** button (pencil icon), click **Add another role**, select the **Service Account Token Creator** role, and click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-grant-role.png) Role based authentication is the preferred authentication mode for GCS based on GCP recommendations, however, providing a service account key to directly log-in to the created service account is an alternative authentication method that can be used if preferred. 1. Back in the **Service accounts** menu, click the Actions dropdown next to the newly created service account and click **Manage keys**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-manage-service-account-keys.png "manage sa keys.png") 2. Click **Add key** and then **Create new key**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-new-key.png "create new key sa.png") 3. Select the **JSON** Key type and click **Create** and make note of the key that is generated. 1. Navigate to the **Cloud Storage** page. 2. Click **Create**. 3. Enter a **bucket name**, choose a **region**. **Note**: at the **Choose how to control access to objects** step, we recommend selecting **Enforce public access prevention on this bucket**. ![](https://storage.googleapis.com/prequel_docs/images/gcs-prevent-public-access.png) 4. After choosing your preferences for the remaining steps, click **Create**. 5. On the **Bucket details** page for the bucket you created, select the **Permissions** tab, and click **Grant access**. 6. Grant access to the principal (Service Account) you created in **Step 1** (*Note: this is the service account you created, not the service account from the prerequisite*), and assign the Roles: **Storage Legacy Bucket Writer**, **Storage Legacy Bucket Reader**, **Storage Legacy Object Reader**. Click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-storage-legacy-bucket-writer.png) Use the following details to complete the connection setup: **bucket name**, your chosen **folder name** for the data, and your **Service account email**. ## Setting up Azure Delta Lake 1. In the Azure portal, navigate to the **Storage accounts** service and click **+ Create**. 2. In the "Basics" tab of the "Create a storage account" form, fill in the required details. 3. In the "Advanced" settings, under "Security" make sure **Enable storage account key access** is turned on. You may turn off (deselect) "Allow enabling public access on containers". Under "Data Lake Storage Gen2", select **Enable hierarchical namespace**. ![](https://storage.googleapis.com/prequel_docs/images/azure-settings-toggles.png "azure settings toggles.png") 4. In the "Networking" settings, you may limit "Network access" to either **Enable public access from all networks** or **Enable public access from selected virtual networks and IP addresses**. If the latter is selected, be sure to add the service's static IP to the address range of the chosen virtual network. All other settings can use the default selections. 5. In the "Data protection" settings, you must turn off **Enable soft delete for blobs**, **Enable soft delete for containers**, and **Enable soft delete for file shares**. ![](https://storage.googleapis.com/prequel_docs/images/azure-turn-off-settings.png "disable default soft delete settings.png") 6. Once the remaining options have been configured to your preference, click **Create**. 1. In the Azure portal, navigate to the **Storage accounts** service and click on the account that was created in the previous step. 2. In the navigation pane, under "Data storage", click **Containers**. Click **+ Container**, choose a name for the container, and click **Create**. 3. In the navigation pane, under "Security + networking", click **Access keys**. 4. Make a note of the **Key** that is generated for either key1 or key2. Use the following details to complete the connection setup: **storage account name**, **container name**, your chosen **folder name** for the data, and your **Access key**. ## Understanding Delta Lake configuration options Changing these attributes on an existing destination table will not take effect until you perform a full refresh of the table. ### Protocol compatibility summary The table below outlines the recommended settings for each feature to qualify for a given protocol minimum reader version. Adjust your configuration to match the protocol version required by your downstream readers. | Protocol MinReader Version | Column Mapping Mode | Deletion Vectors | Change Data Feed | | -------------------------- | ------------------- | ------------------------ | ---------------- | | 1 | NONE | Disabled (Copy-on-Write) | Disabled | | 2 | NONE | Disabled (Copy-on-Write) | Disabled | | 3 | ID | Enabled (Merge-on-Read) | Enabled | *For more details on protocol compatibility, please refer to the [Delta Feature Compatibility documentation](https://docs.delta.io/2.0.2/versioning.html).* There are four key Delta Lake table properties that affect both performance and protocol compatibility. Adjust these settings carefully based on your performance needs and the protocol version supported by your readers. ### `retention_window_days` **Purpose:** Sets the number of days for which historical data (e.g., previous table versions used for time travel or auditing) is retained. **Recommendation:** Set this value according to your organization's internal data retention policies. ### `column_mapping_mode` **Purpose:** Controls how columns are mapped between the underlying storage and the table schema. This setting is critical during schema evolution. **Recommendation:** Set this to `ID` for robust, identifier-based mapping. Use a different setting (such as `NONE` or `NAME`) only if you need to support a lower protocol reader version. ### `deletion_vectors_disabled` **Purpose:** Determines whether deletion vectors are used. * **Deletion Vectors** enable the **merge-on-read** approach, where modifications (like deletes) are applied by marking rows as deleted without rewriting the underlying `Parquet` files. * The traditional **copy-on-write** approach rewrites entire files for each change, which can be slower for small modifications. **Recommendation:** * **Enable deletion vectors** (i.e., set `deletion_vectors_disabled` to `false`) to leverage merge-on-read performance benefits. * **Disable deletion vectors** if you must support a lower minimum reader version. ### `change_data_feed_disabled` **Purpose:** Controls whether the change data feed (CDF) is active. The CDF records row-level changes (inserts, updates, and deletes) for incremental processing, auditing, or real-time analytics. **Recommendation:** * **Keep change data feed enabled** (i.e., set `change_data_feed_disabled` to `false`) by default. * **Disable change data feed** only if you need to support a lower minimum reader version. ## FAQ Delta Lake delivers warehouse-native capabilities such as upserts, time travel, and schema evolution, with the simplicity, scalability, and secure permissions model of an object storage bucket. It gives you the advanced transactional features and data consistency of a data warehouse while eliminating extra compute costs and provisioning required to write directly to a warehouse. This enables your warehouse to be isolated from data sharing, so you can receive data without exposing your internal resources. Delta Lake uses vacuum operations to clean up obsolete data files and maintain transaction isolation. The writer must have delete permissions so that vacuuming can safely remove outdated files without compromising the consistency and isolation of ongoing transactions. Yes, you can direct data to a specific prefix. However, we recommend using a completely isolated bucket to receive data. The Delta Lake destination requires permissions to list objects in the entire bucket, meaning all perms cannot be scoped to a specific prefix. Isolating the destination to a dedicated bucket minimizes security risks and reduces the chance of malformed data mixing with other datasets. No, the data writer is responsible for vacuuming and compacting data as needed. Data consumers should not run any non read queries on the table. To check for updates, you can query the table history. In Spark or Databricks SQL, run: ```sql title="Check table history" icon="database" theme={null} DESCRIBE HISTORY table_name LIMIT 1; ``` This command returns the most recent commit details. Additionally, most bucket providers offer the capability to trigger a webhook or lambda when objects are created. Configure the trigger to execute whenever a file is created in `s3://bucket-name///_delta_log` to know when a table has been updated. * **Merge-on-Read:** * Uses deletion vectors to mark rows as deleted or modified without rewriting entire files. This approach speeds up incremental updates by applying changes during read time. * **Copy-on-Write:** * Rewrites entire `Parquet` files upon any modification, which can be less efficient for small or frequent changes. ## Mounting AWS S3 Delta Lake to Athena **Protocol MinReader Version:** Athena requires delta lake tables compatible with Protocol MinReader Version 1. 1. In the AWS console, navigate to the Athena query editor. 2. Choose the same region as your configured bucket. 3. Execute the following SQL: ```sql title="Mount to Athena" icon="database" expandable theme={null} CREATE EXTERNAL TABLE IF NOT EXISTS schema.table_name.= LOCATION 's3://bucket-name//' TBLPROPERTIES ('table_type' = DELTA); ``` ## Mounting GCS Delta Lake to BigQuery **Protocol MinReader Version:** BigQuery requires delta lake tables compatible with Protocol MinReader Version 3. 1. In the Google Cloud console, navigate to the BigQuery Console. 2. Click the **+ Add Data** button at the top left of the console. 3. Select Google Cloud Storage as a Data Source 4. Select **GCS: (Manual) BigLake External & External Tables: BigQuery** option 5. Set the file format to **Delta**, then write provide the path expression `'//'` 6. Choose a Dataset in the same region as the bucket and a table name. 7. Choose Table Type **External Table**. 8. Create your external table and query data to test. **Schema Evolution:** The external table must be manually refreshed anytime new columns are added. Consult GCP documentation for instructions. ## Mounting AWS S3 Delta Lake to ClickHouse **Managed Credentials:** Clickhouse supports [managed credentials](https://clickhouse.com/docs/integrations/s3#managing-credentials) so that access key or role information does not need to be included in the `CREATE TABLE...` syntax. 1. Open a Clickhouse SQL session. 2. Execute the following SQL: ```sql title="Mount to ClickHouse" icon="database" expandable theme={null} CREATE TABLE schema.table_name AS ENGINE = DeltaLake( 's3://bucket-name//', '', '' ); ``` ## Mounting AWS S3 Delta Lake to DuckDB/MotherDuck **Column Mapping:** DuckDB requires delta lake tables use column mapping mode `NONE`. **Secrets Manager:** DuckDB has a [secrets manager](https://duckdb.org/docs/stable/configuration/secrets_manager.html) which can be used in order for the access key or role information to not need to be included in the `CREATE TABLE...` syntax 1. Install the [DuckDB Delta extension](https://duckdb.org/docs/stable/extensions/delta.html). 2. Open a DuckDB SQL session. 3. Execute the following SQL: ```sql title="Mount to DuckDB" icon="database" expandable theme={null} CREATE VIEW schema.table_name AS SELECT * FROM deltaLake( 's3://bucket-name//', '', '' ); ``` ## Mounting S3 Delta Lake to Databricks (Unity) 1. Ensure your Databricks session has read access to the configured bucket. 2. Open a Databricks SQL session. 3. Execute the following SQL: ```sql title="Mount to Databricks" icon="database" expandable theme={null} CREATE VIEW schema.table_name AS SELECT * FROM delta.`s3a://bucket-name/metastore//` ); ``` ## Mounting S3 Delta Lake to Redshift **Protocol MinReader Version:** Athena requires delta lake tables compatible with Protocol MinReader Version 1. 1. On the AWS Glue console, choose Crawlers in the navigation pane. 2. Choose Create crawler. 3. For Name, enter whatever, and choose "Next". 4. For Data source configuration, choose "Not yet". 5. For Data source, choose Add a data source. 6. For Data source, select Delta Lake. 7. For Include delta lake table paths, enter `s3://bucket-name//` 8. Select Enable write manifest, then choose Add a Delta Lake data source. Choose Next. 9. For IAM role, either select an existing role or create one with permissions to Glue and the bucket 10. For Target database, choose Add database, then Create a database page is shown. 11. For Name, enter whatever, then choose Create database. Then come back to the previous page. For Target database, click the reload button, and select the created database. 12. For Frequency under Crawler schedule, choose double the frequency data is delivered. For example for data arriving on hourly minute frequency, choose 30 minutes. 13. Review your configuration, and choose Create crawler. You can trigger the crawler to run manually via the AWS Glue console, or through the SDK or AWS CLI using the StartCrawl API. You could also schedule a trigger via the AWS Glue console. 14. Wait for the crawler to complete. 15. Navigate to redshift and inspect the awsdatacatalog to find the newly created table ## Reading S3 Delta Lake with PySpark **Broad Compatibility:** Spark can consume Delta Lake tables from S3, Google Cloud Storage, Azure Blob Storage, or other S3 compatible object stores. Delta Lake reads can be done in Java, Python, or Scala Spark. 1. In your PySpark code, run the following to instantiate a dataframe backed by your delta table: ```python title="Read with PySpark" icon="python" theme={null} df = spark.read.format("delta").load("s3://bucket-name//") ``` ## Mounting S3 Delta Lake to Snowflake **Protocol MinReader Version:** Snowflake requires delta lake tables compatible with Protocol MinReader Version 2. 1. Open a Snowflake client. 2. Execute the following SQL, choose your frequency based on desired data freshness: ```sql title="Mount to Snowflake" icon="database" expandable theme={null} CREATE OR REPLACE CATALOG INTEGRATION delta_catalog_name CATALOG_SOURCE = OBJECT_STORE TABLE_FORMAT = DELTA ENABLED = TRUE; CREATE OR REPLACE EXTERNAL VOLUME ext_volume_name STORAGE_LOCATIONS = ( ( NAME = 'some_name' STORAGE_PROVIDER = S3 ... ) ) ALLOW_WRITES = FALSE; CREATE ICEBERG TABLE schema.table_name CATALOG = delta_catalog_name EXTERNAL_VOLUME = ext_volume_name BASE_LOCATION = '/'; CREATE OR REPLACE TASK task_name SCHEDULE = 'USING CRON 0/5 * * * * America/Los_Angeles' -- Runs every 5 minutes, change cron to match desired freshness ... AS ALTER ICEBERG TABLE schema.table_name REFRESH; ``` ## Mounting Delta Lake to Microsoft Fabric & Microsoft OneLake **Column Mapping:** Requires delta lake tables use column mapping mode `NONE`. 1. Navigate to the Microsoft Fabric 2. Navigate to your lakehouse, if you do not have one, create one. 3. In your lakehouse, select the ellipses (…) next to Tables and then select "New shortcut". ![](https://learn.microsoft.com/en-us/fabric/onelake/media/onelake-shortcuts-adb-quickstart/new-table-shortcut.png) 4. In the New shortcut screen, select your bucket provider. Normally this will be Azure Data Lake Storage Gen2 tile. 5. Connect to your bucket and select the path of the delta lake table `''/'` ![](https://learn.microsoft.com/en-us/fabric/onelake/media/onelake-shortcuts-adb-quickstart/connection-details.png) 6. After creating the shortcut, the shortcut should appears as a Delta table under Tables. It may appear as "Unidentified", this is a UI bug in Azure. 7. To confirm the table is correctly mounted, select "SQL Analytics endpoint" from the drop down to the left of the "share" button, in the top right of the console 8. The shortcut should correctly appear as a Delta table under Tables. 9. \[optional] Click the "New semantic model" button at the top of the page to set up the mounted table for use in Microsoft products like Power BI. **Vector Support:** Azure Fabric and Microsoft OneLake do not support Delta Lake vector columns. These columns will be omitted from any table you mount. # GCP SSH tunneling Source: https://docs.prequel.co/export/destinations/gcp-ssh-tunneling Configure a GCP bastion server to accept an SSH connection from our service. **Coming soon!** Contact us for this documentation section. # Google Cloud Storage Source: https://docs.prequel.co/export/destinations/gcs Configuring your Google Cloud Storage destination. ## Prerequisites * By default, GCS authentication uses role-based access. You will need our service account name available to grant access. It should look like `some-name@some-project.iam.gserviceaccount.com`. 1. In the GCP console, navigate to the **IAM & Admin** menu, click into the **Service Accounts** tab, and click **Create service account** at the top of the menu. 2. In the first step, name the service account that will be used to transfer data into Cloud Storage and click **Create and Continue**. Click **Continue** in the following optional step without assigning any roles. 3. In the **Grant users access to this service account** step, within the **Service account users role** field, enter the provided **Service account** (see prerequisite) and click **Done**. 4. Once successfully created, search for the created service account in the service accounts list, click the **Service account** name to view the details, and make a note of the **email** (note: this is a different email than the service's service account). 5. Grant access using one of the following authentication methods: Select the permissions tab, find the provided principal name (**Service account** from the prerequisite), click the **Edit principal** button (pencil icon), click **Add another role**, select the **Service Account Token Creator** role, and click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-grant-role.png) Role based authentication is the preferred authentication mode for Google Cloud Storage based on GCP recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. An HMAC key is a type of credential and can be associated with a service account or a user account to access Google Cloud Storage. 1. Navigate to the **Cloud Storage** page. 2. Click into the **Settings** tab on the left side menu. ![](https://storage.googleapis.com/prequel_docs/images/gcs-bucket-settings.png) 3. Navigate to the **Interoperability** tab and click the **Create a key for a Service Account** button. 4. Select the **Service Account** created in **Step 1**, and click **Create key**. ![](https://storage.googleapis.com/prequel_docs/images/gcs-select-service-account-hmac.png) 5. Make a note of the **Access key** and **Secret**. 1. Navigate to the **Cloud Storage** page. 2. Click **Create**. 3. Enter a **bucket name**, choose a **region**. **Note**: at the **Choose how to control access to objects** step, we recommend selecting **Enforce public access prevention on this bucket**. ![](https://storage.googleapis.com/prequel_docs/images/gcs-prevent-public-access.png) 4. After choosing your preferences for the remaining steps, click **Create**. **Recommendation: dedicated bucket for data transfers** Use a unique bucket for these transfers. This: * Prevents resource contention with other workloads * Avoids accidental data loss from mixed lifecycle or cleanup rules * Improves security by reducing surface area and enabling tighter, destination-scoped policies 5. On the **Bucket details** page for the bucket you created, select the **Permissions** tab, and click **Grant access**. 6. Grant access to the principal (Service Account) you created in **Step 1** (*Note: this is the service account you created, not the service account from the prerequisite*), and assign the Roles: **Storage Object User** and **Storage Legacy Bucket Reader**. Click **Save**. Alternatively, assign a custom role containing at minimum: * `storage.buckets.get` * `storage.objects.create`, `storage.objects.list` The following permissions are recommended, but not required for your custom role: * `storage.objects.get`: enables large files to be uploaded in multiple parallel parts, improving transfer performance. Parts are combined using [GCS object composition](https://cloud.google.com/storage/docs/composing-objects), which requires read access to the temporary part objects. If you grant `storage.objects.get`, you must also grant `storage.objects.delete` so temporary part files are cleaned up and interrupted uploads can be retried. * `storage.objects.delete`: allows connection tests and parallel part uploads to automatically clean up temporary files from the bucket. Your data files themselves are never deleted from the bucket. **Ensure loading is based on file extension** If you enable parallel part uploads, temporary files with a `.part-N` suffix (for example `.parquet.part-0`) are written alongside data files while an upload is in progress and cleaned up when it completes. If a transfer is interrupted before cleanup, part files may remain in the bucket. Any downstream loading must match your chosen file format's extension (`*.parquet`, `*.csv`, `*.json`, or `*.jsonl`) to avoid reading part files as data. Use the following details to complete the connection setup: **bucket name**, your chosen **folder name** for the data, and your **Service account email**. ## Permissions checklist * Service account has been granted **Storage Object User** and **Storage Legacy Bucket Reader** on the bucket, or an equivalent custom role (at minimum `storage.buckets.get`, `storage.objects.create`, and `storage.objects.list`; optionally `storage.objects.get` together with `storage.objects.delete` to enable parallel part uploads, or `storage.objects.delete` alone for connection test cleanup, as described in **Step 2**) * If using service account impersonation, the token creator role is granted to the impersonating principal **Temporary File Cleanup (Optional)** By default, a connection test is performed against the destination during initial configuration. This test writes temporary artifacts under the prefix `_test_connection/` (located at either the bucket root, or inside the custom folder prefix configured on the destination, if present). If you do not grant delete permissions (`storage.objects.delete`), the connection test will still succeed, but these test files will remain in your bucket. To automatically clean them up and avoid clutter or minor storage charges, you can optionally configure **GCS Object Lifecycle Management** on your bucket to delete objects matching the `_test_connection/` prefix (or `/_test_connection/` if a custom folder was configured) after 1 day, or delete them manually. ## FAQ Recommended: use a service account with role-based access (no long-lived user credentials). Optionally, HMAC keys can be used when policy requires, but short-lived tokens and least-privilege roles are preferred. You create one service account in your project with storage permissions, and we use our service account to impersonate yours. This means we never handle your private keys, all operations appear in your audit logs, access is via short-lived tokens, and you can revoke access anytime through your own IAM permissions. Direct service account access is not supported. Data lands in Hive-style partitions per model: `//dt=/_.`. To write to the bucket root, enter `.` as the folder name. `Parquet` (default/recommended), `CSV`, and `JSON`/`JSONL`. Files are automatically split; multiple files may be written per model per transfer. If `storage.objects.get` and `storage.objects.delete` have been granted, large files are also uploaded in multiple parallel parts and combined in the bucket. Temporary part files are named after the data file they belong to, with a `.part-N` suffix appended (for example `.parquet.part-0`), and each contains a raw byte range of that file rather than independently readable data. Part files are visible alongside data files while an upload is in progress and are cleaned up when the upload completes; if a transfer is interrupted, part files may remain in the bucket. Configure any downstream loading to match your chosen file format's extension (`*.parquet`, `*.csv`, `*.json`, or `*.jsonl`) so part files are never read as data. Each transfer writes a manifest file per model under `_manifests`. Files are written per model per transfer in the format: `_manifests//dt=/manifest_{transfer_id}.json`. Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys prioritizing the most recent transfer window; manifest files can help bound the set of files to read. New files are appended to the new location. Existing data remains in the old location. No explicit size/row limits for GCS; files are split automatically based on volume and performance heuristics. # Google Sheets Source: https://docs.prequel.co/export/destinations/google-sheets Configuring your Google Sheets destination. **Google Sheets Destination Limitations** Due to inherent limitations of Google Sheets, this destination is capped at 400,000 rows or 10 million cells for the entire spreadsheet (across all tabs). Any transfer attempting to write more rows to a given sheet will fail. For such cases, we recommend using a different destination instead. 1. Navigate to the your Google Drive or the Google Sheets homepage and create a new Google Sheet in a folder of your choice. 2. In the Google Sheet menu, click **Share** in the top right corner, and enter the Service Account email address provided to you. Grant the Service Account **Editor** permission, and click **Send**. Test your connection and save the destination to complete the connection. During the initial sync, data tables will be loaded as individual tabs, and refreshed at the designated frequency. ## FAQ The integration uses a dedicated service account identity to access your spreadsheet. You grant access by sharing the specific Google Sheet (or a restricted folder) with the generated service account email. Access is controlled by Google Drive sharing permissions (no personal user credentials are required) and can be revoked at any time by removing the service account from the share. # Apache Iceberg Source: https://docs.prequel.co/export/destinations/iceberg Configuring your Apache Iceberg destination. **Beta destination** Apache Iceberg is currently a beta destination with an upper limit of 1 billion rows per month. Please contact our team if you have any questions. ## Setting up Apache Iceberg Apache Iceberg requires a central catalog to manage table metadata and provide atomic transactions. We support several catalog options, each with its own setup guide below: * **[AWS Glue Catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html)** * **[AWS S3 Tables Catalog](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html)** * **[Iceberg REST Catalog](https://iceberg.apache.org/rest-catalog-spec/)** (including R2 Data Catalog and Tabular) * **[Google Lakehouse Catalog](https://cloud.google.com/biglake/docs/lakehouse-iceberg-rest-catalog)** ## Setting up with AWS Glue Catalog **How this works** * The **Glue catalog** stores Iceberg table metadata and the pointer to each table's location. * The **destination S3 bucket** stores your Iceberg data, metadata files, and is used during staging. ### Prerequisites * By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` #### Create bucket 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**. 4. Click **Create bucket**. #### Create policy 1. Navigate to the **IAM** service page. 2. Navigate to the **Policies** navigation tab, and click **Create policy**. 3. Click the **JSON** tab, and paste the following policy, being sure to replace `BUCKET_NAME`, `ACCOUNT_ID`, and `DATABASE` with your specific values. **Why are these permissions necessary?** * The listed **Glue permissions** are needed to manage catalog metadata and handle table operations, including cleaning up temporary tables during syncs. * The listed **S3 permissions** are needed to upload data files, list bucket contents, read Iceberg metadata, and manage files during compaction. * Note: The `glue:CreateDatabase` permission is required if the database does not yet exist. If you wish to use an existing Glue database, you can remove this action and provide us with the name of your pre-existing database. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowGlueAccessToDestinationDatabaseAndTables", "Effect": "Allow", "Action": [ "glue:GetDatabases", "glue:GetDatabase", "glue:GetTables", "glue:GetTable", "glue:GetPartitions", "glue:CreateTable", "glue:CreateDatabase", "glue:UpdateTable", "glue:DeleteTable" ], "Resource": [ "arn:aws:glue:*:ACCOUNT_ID:catalog", "arn:aws:glue:*:ACCOUNT_ID:database/DATABASE", "arn:aws:glue:*:ACCOUNT_ID:database/default", "arn:aws:glue:*:ACCOUNT_ID:table/DATABASE/*" ] }, { "Sid": "AllowS3AccessToBucket", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::BUCKET_NAME", "arn:aws:s3:::BUCKET_NAME/*" ] } ] } ``` **KMS encryption (optional)** If your S3 bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS statement" icon="brackets-curly" theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 4. Click **Next: Tags**, click **Next: Review**. 5. Name the policy, add a description, and click **Create policy**. #### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Use the following details to complete the connection setup: **bucket name**, **bucket region**, **role ARN**, and **Glue database name**. ## Setting up with AWS S3 Tables Catalog **How this works** * The **S3 Tables bucket** stores your Iceberg data and metadata. * A separate **staging S3 bucket** is required for staging data. ### Prerequisites * S3 Tables authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. * The IAM role must also have a trust relationship with itself to function correctly with the S3 Tables API. Your final trust policy should include two principals: our service and the role itself. Be sure to replace `YOUR_ACCOUNT_ID` and `YOUR_ROLE_NAME` with the appropriate identifiers. ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME" }, "Action": "sts:AssumeRole" }, { "Effect": "Allow", "Action": ["sts:AssumeRoleWithWebIdentity"], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` 1. Navigate to the **S3** service page. 2. In the left navigation, click **Table buckets**. 3. Click **Create bucket**. 4. Enter a **Bucket name** and choose the same **AWS Region** you plan to use for your destination S3 bucket. This bucket will be used as your **S3 Tables bucket**. 5. Click **Create bucket**. 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**. 4. Click **Create bucket**. #### Create policy 1. Navigate to the **IAM** service page. 2. Navigate to the **Policies** navigation tab, and click **Create policy**. 3. Click the **JSON** tab, and paste the following policy, replacing `ACCOUNT_ID`, `REGION`, `S3_TABLES_BUCKET_NAME`, and `S3_STAGING_BUCKET_NAME` with the appropriate values. **Why are these permissions necessary?** * The listed **S3 Table data permissions** are needed to read/write Iceberg data files and manage metadata locations in your S3 Tables bucket. * The listed **S3 Table management permissions** are needed to create/manage tables and namespaces (including cleaning up temporary tables during syncs) in your S3 Tables bucket. * The listed **S3 permissions** are needed to write data files to your staging S3 bucket, list bucket contents, and clean up staged or test files. * The permissions to create and manage namespaces (`s3tables:CreateNamespace`, etc.) are required if the namespace does not already exist. If you wish to use an existing namespace, you can remove these actions. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowS3TableDataActions", "Effect": "Allow", "Action": [ "s3tables:GetTable", "s3tables:DeleteTable", "s3tables:GetTableData", "s3tables:PutTableData", "s3tables:GetTableMetadataLocation", "s3tables:UpdateTableMetadataLocation" ], "Resource": "arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/S3_TABLES_BUCKET_NAME/table/*" }, { "Sid": "AllowS3TableManagementAndNamespaceActions", "Effect": "Allow", "Action": [ "s3tables:GetTableBucket", "s3tables:CreateTable", "s3tables:ListTables", "s3tables:CreateNamespace", "s3tables:GetNamespace", "s3tables:ListNamespaces", "s3tables:DeleteNamespace" ], "Resource": "arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/S3_TABLES_BUCKET_NAME" }, { "Sid": "AllowS3AccessToDestinationBucket", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::S3_STAGING_BUCKET_NAME", "arn:aws:s3:::S3_STAGING_BUCKET_NAME/*" ] } ] } ``` **KMS encryption (optional)** If your S3 staging bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS statement" icon="brackets-curly" theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 4. Click **Next: Tags**, click **Next: Review**. 5. Name the policy, and click **Create policy**. #### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy**. Leave the default placeholder trust policy as-is for now (do not paste the final trust policy yet), and click **Next**. The policy must be self-assuming, which is not allowed until the role is created, so it must be updated after the role is created. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name** and click **Create role**. 6. Once successfully created, search for the created role in the Roles list and click the role name. 7. In the role detail view, navigate to the **Trust relationships** tab, click **Edit trust policy**, and replace the default trust policy with the trust policy JSON in the **Prerequisites** section above. Click **Update policy** to save. **AWS IAM Propagation Delay** After updating the trust policy, AWS IAM changes can take **5-10 minutes or longer** to propagate. Please wait for the propagation to complete before testing the connection. Use the following details to complete the connection setup: **S3 Tables bucket ARN**, **destination S3 bucket name**, **destination S3 bucket region**, **role ARN**, and chosen **namespace**. ## Setting up with Iceberg REST Catalog The Iceberg REST catalog is an open standard for interacting with an Iceberg catalog over HTTP. Below are instructions for two popular implementations. **Note on Credentials** We connect to REST catalogs as a standard client using the credentials you provide. We do not support credential vendoring (issuing temporary credentials) for downstream access. ### R2 Data Catalog (Cloudflare) **Tip: Zero Egress Fees** Cloudflare R2 charges no egress fees, making it a cost-effective option if you plan to query your Iceberg data from external locations or other cloud providers. 1. Log in to your Cloudflare dashboard. 2. Follow the [Cloudflare documentation to create an R2 bucket](https://developers.cloudflare.com/r2/buckets/create-buckets/). Make a note of the **Bucket Name** and your **R2 Account ID**. 3. Follow the [Cloudflare documentation to create an R2 API token](https://developers.cloudflare.com/r2/api/tokens/) with **Admin Read & Write** permissions. Make a note of the generated **Access Key ID** and **Secret Access Key**. Use the following details to complete the connection setup: * **Catalog URI**: `https://api.cloudflare.com/client/v4/accounts/YOUR_R2_ACCOUNT_ID/r2/catalog` * **API Token** (as the credential) * **Bucket Name** and **Region** * **R2 Access Key ID** and **R2 Secret Access Key** **R2 catalog path requirement** If you customize the folder or path used for the R2 Data Catalog, it must start with `__r2_data_catalog`. The R2 API does not validate this upfront, so an incorrect prefix will result in runtime failures when creating or querying tables. ### Google BigLake (Lakehouse Catalog) **How this works** * The **Google Lakehouse catalog** stores Iceberg table metadata. * The **destination GCS bucket** stores your Iceberg data, metadata files, and is used during staging. * With Google BigLake, the Iceberg tables become queryable directly from BigQuery. No external table definition or separate mount step is required. #### Prerequisites * You will need a Google Cloud service account with permissions to read/write to your GCS bucket, and manage your Lakehouse catalog. * By default, authentication uses role-based access via service account impersonation. You will need our service account name available to grant access. It should look like `some-name@some-project.iam.gserviceaccount.com`. 1. Navigate to the **Cloud Storage** service page. 2. Click **Create bucket**. 3. Provide a name and choose the appropriate region. Make note of the **Bucket name**. 1. Create a Lakehouse catalog in your Google Cloud Project by following the [Lakehouse catalog documentation](https://docs.cloud.google.com/lakehouse/docs/lakehouse-iceberg-rest-catalog). 2. **Important**: When creating the catalog, ensure that you configure it to use **end-user credentials**. Do NOT use credential vending, as we need to access the underlying storage directly with the provided service account credentials. Ensure your service account has the necessary permissions to access the bucket and catalog. Then, allow the service to impersonate it. #### 1. Grant permissions to your service account You can use preconfigured GCP roles or define a fine-tuned custom role for least-privilege access. **Option A: Preconfigured GCP Roles** Assign the following roles to your service account: * **Storage Object Admin** (`roles/storage.objectAdmin`) on the destination bucket. * **BigLake Editor** (`roles/biglake.editor`) on the target project/catalog to allow creation and management of Iceberg tables via BigLake. **Option B: Fine-Tuned Custom Permissions** For a least-privilege approach, create a custom IAM role with the following individual permissions: **Cloud Storage Permissions** (applied to the destination bucket): * `storage.buckets.get` * `storage.objects.create` * `storage.objects.delete` * `storage.objects.get` * `storage.objects.list` * `storage.objects.update` **BigLake Permissions** (applied to the target project or catalog): * `biglake.catalogs.get` * `biglake.catalogs.list` * `biglake.databases.create` * `biglake.databases.delete` * `biglake.databases.get` * `biglake.databases.list` * `biglake.databases.update` * `biglake.tables.create` * `biglake.tables.delete` * `biglake.tables.get` * `biglake.tables.list` * `biglake.tables.update` #### 2. Allow the service to impersonate your service account Navigate to **IAM & Admin** > **Service Accounts**. Select the service account you just created. Select the **Permissions** tab, click **Grant Access**, enter our service account name (from the prerequisite), and select the **Service Account Token Creator** role. Use the following details to complete the connection setup: **Google Cloud Project ID**, **Catalog Name**, **Schema**, **Bucket Name**, and provide your service account credentials. ### Tabular 1. Log in to your Tabular organization's dashboard. 2. Navigate to the credentials section and create a new credential with permissions to create tables and write data. 3. Make a note of the generated **Client ID** and **Client Secret**. Use the following details to complete the connection setup: * **Catalog URI**: `https://api.tabular.io/ws` * **Client ID** * **Client Secret** ## Understanding Iceberg configuration options Changing these attributes on an existing destination table will not take effect until you perform a full refresh of the table. #### Managing staged data **Purpose:** During each transfer, batches are first written to a staging prefix in your object storage bucket before they are committed into the final Iceberg table. This prefix is always named `_write_ahead_staging` (for example: `your_folder/_write_ahead_staging//` or `_write_ahead_staging//` if no folder/schema is configured). **Recommendation:** We recommend configuring an object storage lifecycle policy to automatically delete objects under the `_write_ahead_staging` prefix after **30 days**. This provides a safety net for any orphaned staged files that are not cleaned up due to failed or interrupted runs. #### retention\_window\_days **Purpose:** Sets the number of days for which historical data (e.g., previous table snapshots used for time travel or auditing) is retained. **Recommendation:** Set this value according to your organization's internal data retention policies. *** ### FAQ We first stage batch files into an object storage bucket, then use your chosen catalog to atomically commit them into the final Iceberg table. For Glue and REST catalogs, the same S3 bucket is used for both staging and permanent table data, with different prefixes. For S3 Tables, batches are staged into your **staging S3 bucket**, then the finalized Iceberg data and metadata is written to the managed **S3 Tables bucket**. There are tradeoffs to consider when choosing between Glue and S3 Tables: * **Glue Catalog**: Glue stores the table metadata, and your S3 bucket stores both staged files and the final Iceberg data under different prefixes. We run snapshot expiry and compaction and you control the S3 layout. This is a good fit if you already use Glue as your central catalog or want to keep data in a single S3 bucket you manage directly. * **S3 Tables Catalog**: The S3 Tables bucket is a fully managed table bucket where AWS stores the finalized Iceberg data and metadata. We write batches to a separate **staging S3 bucket**, and the catalog writes the final data into the S3 Tables bucket and handles maintenance on your behalf. This is a good fit if you prefer automatic maintenance and plan to query through engines that natively support S3 Tables. Apache Iceberg is an open table format designed for analytic datasets on object stores. It delivers warehouse-native capabilities such as ACID transactions, time travel, and schema evolution with the simplicity, scalability, and secure permissions model of an object storage bucket. By using a central catalog, Iceberg provides reliable transactions and enables multiple engines to work concurrently on the same data. This enables your warehouse to be isolated from data sharing, so you can receive data without exposing your internal resources. Iceberg performs background maintenance operations to manage the table's health and performance. These include expiring old snapshots and compacting small data files. The writer must have delete permissions to safely remove obsolete files without compromising data integrity. Yes, you can direct data to a specific prefix (warehouse path). However, we recommend using a completely isolated bucket to receive data. This minimizes security risks and reduces the chance of accidental interference with other datasets. No, the data writer is responsible for expiring snapshots and compacting data as needed. Data consumers should not run any non-read queries on the table, except managed catalogs like R2 and S3 Tables, which automatically run compaction and snapshot expiration. You should treat the destination tables as read-only. Executing write or delete operations manually may corrupt the table state and break data synchronization. You can query the table's metadata to see the history of snapshots. Each snapshot represents a version of the table. For example, in Spark SQL, you can run: ```sql title="Check latest snapshot" icon="database" theme={null} SELECT snapshot_id, committed_at FROM my_glue_catalog.my_db.my_table.snapshots ORDER BY committed_at DESC LIMIT 1; ``` This command returns the most recent snapshot details. Additionally, most bucket providers offer the capability to trigger a webhook or lambda when objects are created, which can be configured to monitor the table's metadata directory for new manifest lists. **Note:** We write a `version-hint.txt` file to the metadata directory. This allows tools like PyIceberg and DuckDB to read the table directly from S3 without needing to connect to the catalog service, by pointing them to the table's root location. We do not enforce size limits, including JSON fields. Downstream query engines may have their own limits (e.g., Amazon Redshift's `SUPER` and `VARCHAR` sizes). Ensure your data fits within your query engine's constraints to avoid query failures. You create one service account in your project with BigLake/Storage permissions, and we use our service account to impersonate yours. This means we never handle your private keys, all operations appear in your audit logs, access is via short-lived tokens, and you can revoke access anytime through your own IAM permissions. Direct service account access is not supported. Yes. Set up the destination with the [Google Lakehouse Catalog](#google-biglake-lakehouse-catalog): tables written through Google BigLake are queryable directly from BigQuery, with no external table definition or separate mount step required. Mounting is not currently supported for the other catalog options. ## Mounting/reading an Iceberg table You can mount/read an Iceberg table into your data warehouse of choice. Below are the supported catalog types and links to the corresponding vendor documentation: * ClickHouse * [Glue catalog](https://clickhouse.com/docs/use-cases/data-lake/glue-catalog) * [S3 Tables catalog](https://clickhouse.com/docs/engines/table-engines/integrations/iceberg) * [REST catalog](https://clickhouse.com/docs/use-cases/data-lake/rest-catalog) * DuckDB / MotherDuck * [Glue catalog](https://duckdb.org/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse) * [S3 Tables catalog](https://duckdb.org/2025/03/14/preview-amazon-s3-tables) * [REST catalog](https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs) * Spark * [Glue catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html#aws-glue-programming-etl-format-iceberg-read-spark) * [S3 Tables catalog](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html) * [REST catalog](https://iceberg.apache.org/docs/latest/spark-configuration/#rest-catalog) * Athena / Redshift * [Glue catalog](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html) * [S3 Tables catalog](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html) * Snowflake * [REST catalog](https://docs.snowflake.com/en/user-guide/tables-iceberg-externally-managed-writes) # Materialize Source: https://docs.prequel.co/export/destinations/materialize Connect a Materialize destination to Prequel. **Coming soon!** To connect a Materialize destination, contact support for the setup guide. # Preparing for Snowflake password deprecation Source: https://docs.prequel.co/export/destinations/migrating-snowflake-authentication-to-key-pair Timeline and migration steps to move Snowflake SERVICE connections to Key Pair authentication. By August to October 2026, all Snowflake `SERVICE` connections must authenticate with Key Pair authentication. In late 2024, Snowflake introduced a new `TYPE` property for the `USER` object. The value determines permitted authentication methods and will become progressively stricter throughout 2025 and 2026. The value can be `null`, `HUMAN`, `SERVICE`, or `LEGACY_SERVICE`. ## Understanding your current connection configuration You can check your user's configuration with: ```sql title="Check user config" icon="database" theme={null} DESCRIBE USER ; -- check TYPE, RSA_PUBLIC_KEY ``` A `TYPE` of `null` is treated like `HUMAN`, and single factor password authentication will eventually be deprecated. **Action required if:** * `TYPE` is `null` or `HUMAN`: Apply either the long-term or temporary fix. * `RSA_PUBLIC_KEY` is set and `TYPE` is `SERVICE`: No action needed. `HUMAN` or `null` users cannot login with single-factor passwords starting August 2026. ## Long term fix: update to key pair authentication **Step 1:** Set the RSA public key (which Prequel provides): ```sql title="Set public key" icon="database" theme={null} ALTER USER SET RSA_PUBLIC_KEY='MIIBI...YOUR_KEY...'; ``` Test your connection to verify it works. **Step 2:** Update the user type: ```sql title="Set user type" icon="database" expandable theme={null} ALTER USER SET TYPE = SERVICE; DESCRIBE USER ; -- verify ALTER USER UNSET PASSWORD; -- optional ``` ## Temporary fix: use LEGACY\_SERVICE type As a short-term workaround, you can set: ```sql title="Set legacy service type" icon="database" theme={null} ALTER USER SET TYPE = LEGACY_SERVICE; ``` However, `LEGACY_SERVICE` users will be deprecated starting in August 2026. # MongoDB Source: https://docs.prequel.co/export/destinations/mongodb Configuring your MongoDB destination. ## Prerequisites * If your MongoDB security posture requires IP whitelisting, have our static IP available during the following steps. It will be required in Step 3. **Network allowlisting** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. ### MongoDB CLI 1. Connect to your **MongoDB node** using the **MongoDB CLI** as an **admin**. 2. Execute the following query: ```javascript title="Get replica set status" icon="code" theme={null} db.adminCommand( { replSetGetStatus : 1 } ).members ``` 3. Make a note of the **host identifiers**. ### MongoDB Atlas 1. Log in to your Atlas **dashboard**, click into the **Database** tab and click **Connect**. 2. Under **Connect your application**, click into the **Drivers** option. 3. In the following screen, make a note of the **host identifier** within your connection string. It can be found within the string that starts with `mongodb+srv` and the identifier is the URI after the `@`. E.g., `some-cluster.some-characters.mongodb.net`. See CLI or Atlas instructions below depending on your MongoDB deployment type. ### MongoDB CLI 1. Connect to your MongoDB node using the MongoDB CLI as an admin. 2. Execute the following script (with a `username` and `password`) to create a new user. Replace `database` with the name of the database you'd like to load data into. ```javascript title="Create user" icon="code" expandable theme={null} use admin db.createUser({ user: "", pwd: "", roles: [ {role: "readWrite", db: ""} ] }) ``` ### MongoDB Atlas 1. Log in to your Atlas **dashboard**, click into the **Database Access** section of the **Security** options. 2. Select **Add New Database User**. Choose the **Password** authentication method, and enter a `username` and `password` for the new user. 3. In the **Database User Privileges** menu, select **Grant Specific User Privileges**. Within **Specific Privileges**, add the following: `readWrite@` (with the name of the database you'd like to load data into). You can leave the **Collection** field blank (`*`). 4. Click **Add User** to create the new user. If your MongoDB instance enforces IP Access restrictions, you'll need to allow access to a static IP. **For self-hosted MongoDB deployments** IP allowlisting should be configured at the network or firewall level (e.g., using cloud provider security groups, `iptables`, or other firewall tools). Please consult your network administrator or your hosting provider's documentation. ### MongoDB Atlas CLI If you manage your Atlas project with the Atlas CLI, execute the following command to add the static IP to your project's IP access list. You may need to append `/32` to the IP address to specify it in CIDR notation. ```bash title="Add IP to access list" icon="terminal" theme={null} atlas accessLists create "/32" --comment "data sharing service static IP" ``` ### MongoDB Atlas 1. Log in to your Atlas **dashboard**, click into the **Network Access** section of the **Security** options. 2. Select **Add IP Address**. In **Access List Entry** enter the static IP of the service. 3. Add an optional comment (e.g., "data sharing service") and click **Confirm**. Use the **host name**, **port** (unless using Atlas, in which case is not needed), and **database name** (as noted in **Step 1**) and the **username** and **password** (as configured in **Step 2**) to complete the connection. ## Permissions checklist * Network: * Inbound rule allows TCP connections from the static egress IP * MongoDB: * `readWrite` on the target database * `atlas accessLists create` (if using Atlas CLI for IP allowlisting) ## FAQ The connection uses a dedicated MongoDB user with `readWrite` role scoped to the target database. Network access can be restricted to the static egress IP. # MotherDuck Source: https://docs.prequel.co/export/destinations/motherduck Configuring your MotherDuck destination. Create a new database for data writing. Skip this step if you already have a database prepared. 1. Log in to the MotherDuck account. 2. Click the plus icon next to "Attached Databases". 3. Enter the desired name of your database. Collect connection information and create an access token for the data transfer service. 1. In the navigation dropdown, select **Settings**. ![](https://storage.googleapis.com/prequel_docs/images/motherduck-endpoints-navigation.png "navigation.png") 2. Click the **Create token** button to create a \*\* access token\*\*. ![](https://storage.googleapis.com/prequel_docs/images/motherduck-endpoints-settings.png "settings.png") 3. Name the token with a descriptive comment and assign the token lifetime. A longer lifetime will ensure you do not have to update the token as often. Ensure that the **Token Type** is set to "Read/Write Token". Click **Generate**. ![](https://storage.googleapis.com/prequel_docs/images/motherduck-endpoints-create-token.png "create-token.png") 4. In the pop up that follows, **copy the token** and securely save the token. Use the following details to complete the connection setup: **database**, your chosen **schema name**, and **access token**. ## Permissions checklist * Access token generated with **Read/Write Token** type. * Token lifetime set according to your rotation policy. * Target database exists in MotherDuck. ## FAQ We authenticate using a MotherDuck access token. Generate a dedicated Read/Write token in MotherDuck Settings and provide it during destination setup. The token grants scoped access without sharing your account credentials. Generate a new token in MotherDuck Settings, update the destination configuration with the new token, and revoke the old one. No. The schema provided in the destination configuration is created automatically on first sync. # Generic MySQL Source: https://docs.prequel.co/export/destinations/mysql Configuring your generic MySQL destination. ## Prerequisites * If your MySQL database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Create a rule in a security group or firewall settings to whitelist: * incoming connections to your host and port (usually `3306`) from the static IP. * outgoing connections from ports `1024` to `65535` to the static IP. **Optional: SSH tunneling** If your database is not accessible from the public internet, SSH tunneling through a bastion host is supported. Allow inbound SSH (port `22`) from the static egress IP on the bastion host, create an SSH user with the service's public key in `~/.ssh/authorized_keys` (contact support for the key), and grant the bastion host's IP access to the database port in place of the static egress IP. Provide the bastion host address, port, and username in the destination configuration. Create a database user to perform the writing of the source data. 1. Open a connection to your MySQL database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create user" icon="database" theme={null} CREATE USER @'%' IDENTIFIED BY ''; ``` **Credential character limitations** For user credentials containing special characters, please avoid using the following characters: `@`, `[`, `]`, `/`, `?`, `#`, `"`, `\\`, `+`, space, `&`, `:`, `%` as these characters can break connection string parsing. 3. Grant user required privileges on the database. ```sql title="Grant privileges" icon="database" theme={null} GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW ON *.* TO @'%'; ``` **If the `schema`/`database` already exists** By default, the service creates a new schema (*in MySQL, `schema` is synonomous with `database`*). If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using `GRANT ALL PRIVILEGES ON .* TO @'%';` Use the following details to complete the connection setup: **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password**. ## Permissions checklist * Database user has `SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, CREATE VIEW` on `*.*`. * If using a pre-created schema: user has `GRANT ALL PRIVILEGES ON .*`. * Firewall or security group allows the service's egress IP on port 3306. If using SSH tunneling, allow the egress IP on port 22 on the bastion host instead. ## FAQ We connect using the credentials you provide (host, port, username, password) over TCP. If your database is not publicly accessible, SSH tunneling through a bastion host is supported. The service uses public key authentication for bastion access. MySQL 8.0 and above are fully supported. MySQL 5.7 is best-effort only. No. The schema provided in the destination configuration is created automatically on first sync. If you pre-create it, grant `ALL PRIVILEGES ON .*` to the writer user. # Oracle Source: https://docs.prequel.co/export/destinations/oracle Configuring your Oracle destination. ## Prerequisites * If your Oracle database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Create a rule in a security group or firewall settings to whitelist: * incoming connections to your host and port (usually `1521`) from the static IP. * outgoing connections from ports `1024` to `65535` to the static IP. **Optional: SSH tunneling** If your database is not accessible from the public internet, SSH tunneling through a bastion host is supported. Allow inbound SSH (port `22`) from the static egress IP on the bastion host, create an SSH user with the service's public key in `~/.ssh/authorized_keys` (contact support for the key), and grant the bastion host's IP access to the database port in place of the static egress IP. Provide the bastion host address, port, and username in the destination configuration. Create a database user to perform the writing of the source data. 1. Open a connection to your Oracle database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create user" icon="database" theme={null} CREATE USER IDENTIFIED BY ''; GRANT CREATE SESSION TO ; ``` 3. Grant user required privileges on the database. ```sql title="Grant privileges" icon="database" expandable theme={null} GRANT CREATE SESSION TO ; GRANT ALTER USER TO ; GRANT CREATE ANY TABLE TO ; GRANT CREATE ANY INDEX TO ; GRANT SELECT ANY TABLE TO ; GRANT INSERT ANY TABLE TO ; GRANT UPDATE ANY TABLE TO ; GRANT DELETE ANY TABLE TO ; GRANT DROP ANY TABLE TO ; GRANT COMMENT ANY TABLE TO ; ``` **If the `schema`/`database` already exists** By default, the service creates a new schema (*in Oracle, `schema` is synonomous with `user`*). If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema.\` Use the following details to complete the connection setup: **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password**. ## Permissions checklist * User has `CREATE SESSION`. * User has `CREATE ANY TABLE, CREATE ANY INDEX, SELECT ANY TABLE, INSERT ANY TABLE, UPDATE ANY TABLE, DELETE ANY TABLE, DROP ANY TABLE, COMMENT ANY TABLE`. * Firewall or security group allows the service's egress IP on port 1521. If using SSH tunneling, allow the egress IP on port 22 on the bastion host instead. ## FAQ We connect using the credentials you provide (host, port, username, password) over TCP. If your database is not publicly accessible, SSH tunneling through a bastion host is supported. The service uses public key authentication for bastion access. No. The schema provided in the destination configuration is created automatically on first sync. If you pre-create it, ensure the writer user has the proper permissions on the existing schema. # Overview Source: https://docs.prequel.co/export/destinations/overview Vendor-specific setup guides for all supported destination types. Select your destination vendor below for connection setup instructions. For an overview of how destinations work in Prequel, see [Destinations](/export/concepts/destinations). ## Supported destinations You can find all currently supported destinations and documentation links below. You also may decide to self-host the destination configuration instructions on your own documentation site. If you prefer to do that, we maintain a copy of the source `markdown` files accessible at public locations below: | Vendor | Type | Status | Docs | Markdown (`.md` file) | | :-------------------- | :------------- | :----- | :----------------------------------------------- | :---------------------------------------------------------------------------------------------------- | | `snowflake` | OLAP | GA | [link](/export/destinations/snowflake) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-snowflake.md) | | `bigquery` | OLAP | GA | [link](/export/destinations/bigquery) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-bigquery.md) | | `redshift` | OLAP | GA | [link](/export/destinations/redshift) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-redshift.md) | | `redshift_serverless` | OLAP | GA | [link](/export/destinations/redshift-serverless) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-redshift_serverless.md) | | `databricks` | OLAP | GA | [link](/export/destinations/databricks) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-databricks.md) | | `athena` | OLAP | GA | [link](/export/destinations/athena) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-athena.md) | | `clickhouse` | OLAP | GA | [link](/export/destinations/clickhouse) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-clickhouse.md) | | `motherduck` | OLAP | Beta | [link](/export/destinations/motherduck) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-motherduck.md) | | `delta_lake` | OTF | GA | [link](/export/destinations/delta-lake) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-delta_lake.md) | | `iceberg` | OTF | Beta | [link](/export/destinations/iceberg) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-iceberg.md) | | `postgres` | OLTP | GA | [link](/export/destinations/postgres) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-postgres.md) | | `aurora_postgres` | OLTP | GA | [link](/export/destinations/postgres) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-aurora_postgres.md) | | `mysql` | OLTP | GA | [link](/export/destinations/mysql) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-mysql.md) | | `aurora_mysql` | OLTP | GA | [link](/export/destinations/aurora-mysql) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-aurora_mysql.md) | | `mongodb` | OLTP | GA | [link](/export/destinations/mongodb) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-mongodb.md) | | `sql_server` | OLTP | GA | [link](/export/destinations/sql-server) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-sql_server.md) | | `oracle` | OLTP | Beta | [link](/export/destinations/oracle) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-oracle.md) | | `singlestore` | OLTP | Beta | | | | `s3` | Object Storage | GA | [link](/export/destinations/s3) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-s3.md) | | `s3_compatible` | Object Storage | GA | [link](/export/destinations/s3-compatible) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-s3_compatible.md) | | `gcs` | Object Storage | GA | [link](/export/destinations/gcs) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-gcs.md) | | `abs` | Object Storage | GA | [link](/export/destinations/abs) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-abs.md) | | `sftp` | Object Storage | GA | [link](/export/destinations/sftp) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-sftp.md) | | `google_sheets` | Spreadsheet | GA | [link](/export/destinations/google-sheets) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-google_sheets.md) | ## Other available guides | Guide | Type | Status | Docs | Docs (`.md` file) | | :---------------------------------- | :--------------- | :----- | :------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------- | | S3 Staging Bucket | Staging Resource | GA | [link](/export/destinations/staging-s3) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-staging-s3.md) | | Google Cloud Storage Staging Bucket | Staging Resource | GA | [link](/export/destinations/staging-gcs) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-staging-gcs.md) | | Azure Blob Storage Staging Bucket | Staging Resource | GA | [link](/export/destinations/staging-abs) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-staging-abs.md) | | AWS to GCP role assumption | Authentication | GA | [link](/export/destinations/aws-to-gcp-role-assumption) | [link](https://storage.googleapis.com/prequel_docs/documentation/destinations-aws-to-gcp-role-assumption.md) | You should know You can use your discretion to decide what documentation to provide to your users that wish to connect their destination. To avoid confusion, we recommend working with your user to determine which database or data warehouse destination type they wish to connect, and then sending over the subset of documentation for them to work with. ## Format of landed data ### Data warehouses & databases *(incl. Snowflake, BigQuery, Redshift, Databricks)* Data transferred to data warehouses and relational databases will be loaded as properly typed tables within a single schema. A [`_transfer_status`](/export/features/transfer-status-table) artifact is created in the destination schema to record transfer metadata: specifically, a `transfer_last_updated_at` timestamp for each model. For most destinations this is a table. For BigQuery, it is a view backed by `__TABLES__` metadata and a `_transfer_status_cache` table. ### Object storage *(incl. AWS S3, Google Cloud Storage, Azure blob storage)* Data transferred to object storage destinations will be loaded as Apache `Parquet`, `CSV`, or `JSON` files in Apache Hive style partitions. The default file format is `Parquet`, which we strongly recommend for performance reasons. This means data will appear in the following folder structure: `///dt=/_.parquet` Where: * `` and `` are provided during destination configuration. * `` is the name of the data model being transferred (this is equivalent to a table name in relational data destinations). * `` and `` are generated at transfer time and based on the transfer's start time. `` is of the form `2006-01-01`, while `` is of the form `20060102150405`. * `` is a monotonically increasing integer for a given timestamp, and does not carry any special meaning. Prequel also provides a **manifest file** for each model within each transfer, mapping each transfer to its file contents. See [Manifest files for object storage](/export/features/manifest-files-for-object-storage) for details on the file structure. What are Apache Hive style partitions and Apache Parquet file format? * **Apache Hive style partitions** are compatible with most popular query engines, and should make data easily queryable and transportable. * **Apache Parquet file format** is an open source, column-oriented data file format that offers efficient data compression and data integrity. Note that individual files for object storage destinations will not exceed \~4 GB in size. Rows transferred to object storage destinations are treated as "append only". This has the following effects for all object storage destinations: * Prequel will always write data based on the current data model, meaning **Schema evolution** and **data model updates** must be handled downstream of transfer. * **Deduplication**, **merging updated rows** will not take place during transfer, all updates will be written as unique rows. These updates must be also be made downstream of transfer. * **Full refresh** transfers will append the full dataset without deleting existing data. However, object storage locations configured as a [Delta Lake](/export/destinations/delta-lake) destination **will** receive the native **schema evolution**, **deduplication**, and **ACID transaction** behavior that all data warehouse destinations receive. ### Spreadsheets *(incl. Google Sheets)* Data transferred to spreadsheet destinations will be loaded as a newly created tab per data model. Where possible, the tabs will be created as protected tabs (or "read-only") to prevent accidental modification. # Generic Postgres Source: https://docs.prequel.co/export/destinations/postgres Configuring your generic Postgres destination. ## Prerequisites * If your Postgres database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Create a rule in a security group or firewall settings to whitelist: * incoming connections to your host and port (usually `5432`) from the static IP. * outgoing connections from ports `1024` to `65535` to the static IP. **Optional: SSH tunneling** If your database is not accessible from the public internet, SSH tunneling through a bastion host is supported. Allow inbound SSH (port `22`) from the static egress IP on the bastion host, create an SSH user with the service's public key in `~/.ssh/authorized_keys` (contact support for the key), and grant the bastion host's IP access to the database port in place of the static egress IP. Provide the bastion host address, port, and username in the destination configuration. Create a database user to perform the writing of the source data. 1. Open a connection to your PostgreSQL database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD ''; ``` **Credential character limitations** For user credentials containing special characters, please avoid using the following characters: `@`, `[`, `]`, `/`, `?`, `#`, `"`, `\\`, `+`, space, `&`, `:`, `%` as these characters can break connection string parsing. 3. Grant user `create` and `temporary` privileges on the database. `create` allows the service to create new schemas and `temporary` allows the service to create temporary tables. ```sql title="Grant privileges" icon="database" theme={null} GRANT CREATE, TEMPORARY ON DATABASE TO ; ``` **If the `schema` already exists** By default, the service creates a new schema based on the destination configuration (in the next step). If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using `GRANT ALL ON schema TO ;` Use the following details to complete the connection setup: **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password**. ## Permissions checklist * Database user has `CREATE` and `TEMPORARY` on the target database. * If using a pre-created schema: user has `GRANT ALL ON SCHEMA `. * Firewall or security group allows the service's egress IP on port 5432. If using SSH tunneling, allow the egress IP on port 22 on the bastion host instead. ## FAQ We connect using the credentials you provide (host, port, username, password) over TCP. If your database is not publicly accessible, SSH tunneling through a bastion host is supported. The service uses public key authentication for bastion access. PostgreSQL 13 and above are fully supported. PostgreSQL 12 is best-effort only. No. The schema provided in the destination configuration is created automatically on first sync. If you pre-create it, grant `ALL` on the schema to the writer user and you may remove the database-level `CREATE` permission (retain `TEMPORARY`). # GCP Postgres Source: https://docs.prequel.co/export/destinations/postgres-gcp Configuring your GCP Postgres destination. **Coming soon!** To connect a Postgres (GCP) destination, contact support for the setup guide. # Redshift Source: https://docs.prequel.co/export/destinations/redshift Configuring your Redshift destination. ## Prerequisites * If your Redshift security posture requires IP whitelisting, have our static IP available during the following steps. It will be required in Step 2. * By default, Redshift authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` **Network allowlisting** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. 1. Connect to Redshift using the SQL client. 2. Execute the following query to create a user to write the data (replace `` with a password of your choice). ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD ''; ``` **Creating a user without a password.** Role based auth does not require a password. You may create the user using `CREATE USER PASSWORD DISABLE;`. 3. Grant user `create` and `temporary` privileges on the database. `create` allows the service to create new schemas and `temporary` allows the service to create temporary tables. ```sql title="Grant privileges" icon="database" theme={null} GRANT CREATE, TEMPORARY ON DATABASE TO ; ``` **The schema will be created during the first sync** The schema name supplied as part of Step 4 will be created during the first connection. It does not need to be created manually in the destination ahead of time. **If the `schema` already exists** By default, the service creates a new schema based on the destination configuration. If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using `GRANT ALL ON schema TO ;` Once you've provided the `GRANT ALL` permission on the schema, you can safely remove the `CREATE` permission on the database (but you must retain the `TEMPORARY` permission on the database). 1. In the Redshift console, click **Clusters**, and make a note of the **cluster** name. 2. Select the cluster you would like to connect. 3. In the **General information** pane, make note of the **Endpoint** details. You may need to use the **copy** icon to copy the full details to discover the full endpoint and port number. ![](https://storage.googleapis.com/prequel_docs/images/redshift-endpoint-details.png "redshift endpoint details.png") 4. Click the **Properties** tab. 5. Scroll down to the **Network and security settings** section. 6. In the VPC security group field, select a security group to open it. ![](https://storage.googleapis.com/prequel_docs/images/redshift-vpc-security-groups.png "redshift vpc s groups.png") 7. In the Security Groups window, click **Inbound rules**. 8. Click **Edit inbound rules**. 9. In the Edit the Inbound rules window, follow the steps below to create custom TCP rules for the static IP: a. Select **Custom TCP** in the drop-down menu. b. Enter your Redshift port number. (likely `5439`) c. Enter the **static IP**. d. Click **Add rule**. **Public accessibility and subnet requirements** For IP allowlisting from outside your VPC, the Redshift cluster must be set to **Publicly accessible** and deployed in a **public subnet** with a route to an Internet Gateway. For private Redshift clusters, SSH tunneling is supported. Contact the team for instruction on configuring an SSH tunnel for your Redshift cluster. ### Create staging bucket 1. Navigate to the S3 service page. 2. Click Create bucket. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "**ACLs disabled**" and **Block Public Access settings for this bucket** can be set to "**Block all public access**" as recommended by AWS. Make note of the Bucket name and AWS Region. 4. Click **Create bucket**. **Optional: Add a short retention lifecycle policy** You may configure a lifecycle rule on the staging bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. In the bucket **Management** tab, click **Create lifecycle rule**, set an expiration action for current versions of objects with a 2-day age. Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ### Create policy 1. Navigate to the **IAM** service page, click on the **Policies** navigation tab, and click **Create policy**. 2. Click the JSON tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen above, and `REGION_NAME`, `ACCOUNT_ID`, `CLUSTER_NAME`, `USERNAME`, and `DATABASE_NAME` with the proper Redshift values. 1. **Note**: the first bucket permission in the list applies to `BUCKET_NAME` whereas the second permission applies only to the bucket's contents (`BUCKET_NAME/*`), an important distinction. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" }, { "Effect": "Allow", "Action": "redshift:GetClusterCredentials", "Resource": [ "arn:aws:redshift:REGION_NAME:ACCOUNT_ID:dbuser:CLUSTER_NAME/USERNAME", "arn:aws:redshift:REGION_NAME:ACCOUNT_ID:dbname:CLUSTER_NAME/DATABASE_NAME" ] } ] } ``` **KMS encryption (optional)** If your S3 staging bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS policy statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. **Credential character limitations** For user credentials containing special characters, please avoid using the following characters: `@`, `[`, `]`, `/`, `?`, `#`, `"`, `\\`, `+`, space, `&`, `:`, `%` as these characters can break connection string parsing. 3. Click through to the **Review** step, choose a **name** for the policy, for example, `transfer-service-policy` (this will be referenced in the next step), add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy (from the prerequisite) to allow AssumeRole access to this role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Role based authentication is the preferred authentication mode for Redshift based on AWS recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. 1. Navigate to the **IAM** service page. 2. Navigate to the **Users** navigation tab, and click **Add users**. 3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**. 4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**. 5. Click **Next: Review** and click **Create user**. 6. In the **Success** screen, record the **Access key ID** and the **Secret access key**. Use the following details to complete the connection setup: **username**, **host**, **database**, **cluster**, your chosen **schema**, **IAM role ARN**, and **staging bucket details**. ## Permissions checklist * Redshift database user exists and has `CREATE` and `TEMPORARY` on the database. If you pre-created the schema, ensure `GRANT ALL ON SCHEMA TO `. * IAM role trust policy allows us to assume the role. * IAM policy includes: * `redshift:GetClusterCredentials` on your target cluster (db user and db name resources). * S3 `ListBucket` on `arn:aws:s3:::BUCKET_NAME`. * S3 `GetObject`, `PutObject`, `DeleteObject` on `arn:aws:s3:::BUCKET_NAME/*`. * Network allowlisting (if enforced) permits egress IP/CIDR for the Redshift port (typically 5439). ## FAQ We use role-based authentication with your AWS IAM Role. We assume your role to obtain short-lived database credentials and network access can be constrained by allowlisting the static egress IPs noted above. Redshift's high-throughput path loads data from S3 using `COPY`. We stage files briefly in your bucket to maximize throughput and reliability. Files are cleaned up after load. These are identity claims used in the IAM trust policy when federating from GCP to AWS. `sub` uniquely identifies our Google principal in federation. `oaud` is an additional claim used to bind role assumption to your organization. Common causes: * Missing or incorrect permission on `redshift:GetClusterCredentials` (ensure it targets the correct cluster ARN and region/account). * Trust policy mismatch (our principal isn't permitted to assume your role). * Using a Serverless workgroup permission or `redshift-serverless:GetCredentials` instead of provisioned cluster + `redshift:GetClusterCredentials`. * Propagation delay: IAM changes can take a few minutes to apply. Retry after 5-10 minutes. No. The schema provided in the destination configuration is created automatically on first sync. If you pre-create it, grant `ALL` on the schema to the writer user and you may remove the database-level `CREATE` permission (retain `TEMPORARY`). # Redshift Serverless Source: https://docs.prequel.co/export/destinations/redshift-serverless Configuring your Redshift Serverless destination. ## Prerequisites * If your Redshift security posture requires IP whitelisting, have our static IP available during the following steps. It will be required in Step 2. * By default, Redshift authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` **Network allowlisting** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. ## How authentication works **Two identities involved** **AWS IAM Role (in your AWS account)** You create this role in Step 3 and enter its ARN in the destination form. It holds the S3 staging permissions and allows us to call `redshift-serverless:GetCredentials` on your workgroup. We assume it to obtain short-lived credentials. **Redshift database user (in your Redshift workgroup)** Redshift derives this user from the IAM identity used to authenticate. You create this user and grant it privileges in Step 1. The database user is `IAMR:`, where `` is the final segment of the IAM role ARN entered in the destination form. The ARN `arn:aws:iam::123456789012:role/transfer-role` connects as `IAMR:transfer-role`. The database user is `IAM:`, where `` is the name of the IAM user that owns the access keys. Connections are made as a database user named after your IAM identity. Decide now on the name of the IAM role you will create in Step 3 (for example, `transfer-role`). The database user is that role name prefixed with `IAMR:`. 1. Connect to Redshift using the SQL client. 2. Execute the following query to create the user for your auth method. The quotes are required because the name contains a colon. The name must exactly match `IAMR:` followed by the role name from Step 3. ```sql title="Create user" icon="database" theme={null} CREATE USER "IAMR:" PASSWORD DISABLE; ``` The name must exactly match `IAM:` followed by the name of the IAM user that owns the access keys. Use this name in place of `IAMR:` throughout this step. ```sql title="Create user" icon="database" theme={null} CREATE USER "IAM:" PASSWORD DISABLE; ``` **User already exists?** Redshift creates this user automatically the first time we connect. If a connection attempt has already been made, the `CREATE USER` statement will fail with a "user already exists" error. Skip it and run the grant below. 3. Grant the user `create` and `temporary` privileges on the database. `create` allows the service to create new schemas and `temporary` allows the service to create temporary tables. ```sql title="Grant privileges" icon="database" theme={null} GRANT CREATE, TEMPORARY ON DATABASE TO "IAMR:"; ``` **The schema will be created during the first sync** The schema name supplied as part of Step 4 will be created during the first connection. It does not need to be created manually in the destination ahead of time. **If the `schema` already exists** By default, the service creates a new schema based on the destination configuration. If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using `GRANT ALL ON schema TO "IAMR:";` Once you've provided the `GRANT ALL` permission on the schema, you can safely remove the `CREATE` permission on the database (but you must retain the `TEMPORARY` permission on the database). 1. In the Redshift console, click **Workgroups**, and make a note of the **workgroup** name. 2. Select the workgroup you would like to connect. 3. In the **General information** pane, make note of the **Endpoint** details. You may need to use the **copy** icon to copy the full details to discover the full endpoint and port number. ![](https://storage.googleapis.com/prequel_docs/images/redshift-endpoint-details.png "redshift endpoint details.png") 4. Click the **Properties** tab. 5. Scroll down to the **Network and security settings** section. 6. In the VPC security group field, select a security group to open it. ![](https://storage.googleapis.com/prequel_docs/images/redshift-vpc-security-groups.png "redshift vpc s groups.png") 7. In the Security Groups window, click **Inbound rules**. 8. Click **Edit inbound rules**. 9. In the Edit the Inbound rules window, follow the steps below to create custom TCP rules for the static IP: a. Select **Custom TCP** in the drop-down menu. b. Enter your Redshift port number. (likely `5439`) c. Enter the **static IP**. d. Click **Add rule**. ### Create staging bucket 1. Navigate to the S3 service page. 2. Click Create bucket. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "**ACLs disabled**" and **Block Public Access settings for this bucket** can be set to "**Block all public access**" as recommended by AWS. Make note of the Bucket name and AWS Region. 4. Click **Create bucket**. **Optional: Add a short retention lifecycle policy** You may configure a lifecycle rule on the staging bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. In the bucket **Management** tab, click **Create lifecycle rule**, set an expiration action for current versions of objects with a 2-day age. Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ### Create policy 1. Navigate to the **IAM** service page, click on the **Policies** navigation tab, and click **Create policy**. 2. Click the JSON tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen above, and `REGION_NAME`, `ACCOUNT_ID`, and `WORKGROUP_NAME_OR_ID` with the proper Redshift Serverless values. 1. **Note**: the first bucket permission in the list applies to `BUCKET_NAME` whereas the second permission applies only to the bucket's contents (`BUCKET_NAME/*`), an important distinction. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "redshift-serverless:GetCredentials" ], "Resource": [ "arn:aws:redshift-serverless:REGION_NAME:ACCOUNT_ID:workgroup/WORKGROUP_NAME_OR_ID" ] }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` **KMS encryption (optional)** If your S3 staging bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS policy statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 3. Click through to the **Review** step, choose a **name** for the policy, for example, `transfer-service-policy` (this will be referenced in the next step), add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy (from the prerequisite) to allow AssumeRole access to this role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Role based authentication is the preferred authentication mode for Redshift based on AWS recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. 1. Navigate to the **IAM** service page. 2. Navigate to the **Users** navigation tab, and click **Add users**. 3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**. 4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**. 5. Click **Next: Review** and click **Create user**. 6. In the **Success** screen, record the **Access key ID** and the **Secret access key**. Use the following details to complete the connection setup: **workgroup**, **host**, **database**, your chosen **schema**, **IAM role ARN** (or access keys), and **staging bucket details**. The final segment of the IAM role ARN must match the role name used for the database user in Step 1. ## Permissions checklist * Redshift database user `"IAMR:"` (or `"IAM:"` for access key authentication) exists and has `CREATE` and `TEMPORARY` on the database. If you pre-created the schema, ensure `GRANT ALL ON SCHEMA TO "IAMR:"`. * IAM role trust policy allows us to assume the role. * IAM policy includes: * `redshift-serverless:GetCredentials` on your target `workgroup` ARN (Serverless uses workgroups, not clusters). * S3 `ListBucket` on `arn:aws:s3:::BUCKET_NAME`. * S3 `GetObject`, `PutObject`, `DeleteObject` on `arn:aws:s3:::BUCKET_NAME/*`. * Network allowlisting (if enforced) permits egress IP/CIDR for the Redshift port (typically 5439). ## FAQ We use role-based authentication with your AWS IAM Role. We assume your role to obtain short-lived database credentials and network access can be constrained by allowlisting the static egress IPs noted above. Redshift's high-throughput path loads data from S3 using `COPY`. We stage files briefly in your bucket to maximize throughput and reliability. Files are cleaned up after load. We require `ListBucket` to enumerate staged files and `GetObject/PutObject/DeleteObject` to write, read back, and delete staged files. In Serverless, temporary database credentials are issued per workgroup. Granting this action on the target workgroup allows our assumed role to obtain ephemeral credentials for the database user without long-lived secrets, improving security and auditability. These are identity claims used in the IAM trust policy when federating from GCP to AWS. `sub` uniquely identifies our Google principal in federation. `oaud` is an additional claim used to bind role assumption to your organization. Common causes: * Missing or incorrect permission on `redshift-serverless:GetCredentials` (ensure it targets the correct workgroup ARN and region/account). * Trust policy mismatch (our principal isn't permitted to assume your role). * Using a Cluster ARN or `redshift:GetClusterCredentials` instead of Serverless `workgroup` + `redshift-serverless:GetCredentials`. * Database permission errors: privileges were granted to a manually created user instead of the auto-generated `"IAMR:"` user. Grant privileges to the IAM-derived user described in Step 1. * Propagation delay: IAM changes can take a few minutes to apply. Retry after 5-10 minutes. No. The schema provided in the destination configuration is created automatically on first sync. If you pre-create it, grant `ALL` on the schema to the writer user and you may remove the database-level `CREATE` permission (retain `TEMPORARY`). # S3 Source: https://docs.prequel.co/export/destinations/s3 Configuring your AWS S3 destination. ## Prerequisites * By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:oaud": "", "accounts.google.com:sub": "" } } } ] } ``` ### Create bucket 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**. 4. Click **Create bucket**. **Recommendation: dedicated bucket for data transfers** Use a unique bucket for these transfers. This: * Prevents resource contention with other workloads * Avoids accidental data loss from mixed lifecycle or cleanup rules * Improves security by reducing surface area and enabling tighter, destination-scoped policies ### Create policy 1. Navigate to the **IAM** service page. 2. Navigate to the **Policies** navigation tab, and click **Create policy**. 3. Click the **JSON** tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen in Step 1. ```json title="IAM policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject"], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` **Connection Test File Cleanup (Optional)** By default, a connection test is performed against the destination during initial configuration. This test writes temporary artifacts under the prefix `_test_connection/` (located at either the bucket root, or inside the custom folder prefix configured on the destination, if present). If you do not grant delete permissions (`s3:DeleteObject`), the connection test will still succeed, but these test files will remain in your bucket. To automatically clean them up and avoid clutter or minor storage charges, you can optionally configure an **S3 Lifecycle Policy** on your bucket to expire objects matching the `_test_connection/` prefix (or `/_test_connection/` if a custom folder was configured) after 1 day, or delete them manually. **KMS encryption (optional)** S3 destinations support buckets with KMS encryption (CMK). Encryption with SSE-C is not currently supported. For KMS encryption, add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key: ```json title="KMS policy statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 4. Click **Next: Tags**, click **Next: Review**. 5. Name the policy, add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Role based authentication is the preferred authentication mode for S3 based on AWS recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. 1. Navigate to the **IAM** service page. 2. Navigate to the **Users** navigation tab, and click **Add users**. 3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**. 4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**. 5. Click **Next: Review** and click **Create user**. 6. In the **Success** screen, record the **Access key ID** and the **Secret access key**. Use the following details to complete the connection setup: **bucket name**, **bucket region**, and **role ARN**. ## Permissions checklist * IAM policy on the role allows: * `s3:PutObject` on `arn:aws:s3:::BUCKET_NAME/*` * *(Optional)* `s3:DeleteObject` on `arn:aws:s3:::BUCKET_NAME/*` (if you wish to let connection tests automatically clean up test files instead of using a lifecycle policy) * If using KMS encryption (CMK), IAM policy also allows: * `kms:GenerateDataKey` and `kms:Decrypt` on your CMK ARN * Bucket exists in the intended region; folder prefix (if any) is configured as desired * Trust policy allows the data transfer service to assume the role ## FAQ The recommended approach is role-based access using an IAM Role with a scoped permissions policy. The role is assumed via a trust policy and short-lived credentials, so no long-lived access keys are required. Optionally, access can be configured with HMAC access keys if your policies require it. For at-rest encryption, S3-managed encryption or KMS CMKs are supported (see the KMS callout above for required actions). Grant only the minimum permissions needed (PutObject, and DeleteObject for initial connection test). These are identity claims used in the IAM trust policy when federating from GCP to AWS. `sub` uniquely identifies our Google principal in federation. `oaud` is an additional claim used to bind role assumption to your organization. Data lands in Hive-style partitions per model: `//dt=/_.`. You can set `` during configuration. `Parquet` (default/recommended), `CSV`, and `JSON`/`JSONL`. Files are automatically split; multiple files may be written per model per transfer. Each transfer writes a manifest file per model under `_manifests`. The `_manifests` folder is created automatically at the root of the bucket. Files are written per model per transfer in the following format: `_manifests//dt=/manifest_{transfer_id}.json`. Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys prioritizing the most recent transfer window; manifest files can help bound the set of files to read. # S3 compatible Source: https://docs.prequel.co/export/destinations/s3-compatible Configuring your S3 compatible destination. ## Prerequisites * Many object storage platforms offer "S3 compatibility" enabling writing to and reading from the object storage using the S3 protocol. The S3 protocol uses an HMAC key comprised of an access ID and a secret to authenticate and write data. Consult your object storage platform's documentation to learn how to generate an HMAC Access ID and Secret. Use the following details to complete the connection setup: **bucket host**, **bucket name**, chosen **folder name**, **HMAC access ID**, and **HMAC secret**. ## Permissions checklist * HMAC Access ID and Secret have write permissions to the target bucket. * Bucket is accessible from the service's egress IP. ## FAQ We authenticate using the HMAC access ID and secret you provide. These credentials are used to sign requests to the S3-compatible API. Scope the credentials to only the required bucket to limit access. `Parquet` (default/recommended), `CSV`, `JSON`, and `JSONL`. Files are automatically split; multiple files may be written per model per transfer. Data lands in Hive-style partitions per model: `//dt=/_.`. Multiple files may be written per model per transfer. Manifest files are written to a `_manifests/` directory at the root of the configured folder path. Generate new HMAC credentials from your object storage platform and update them in the destination configuration. # SFTP Source: https://docs.prequel.co/export/destinations/sftp Configuring your SFTP server. ## Prerequisites * By default, SFTP uses keypair authentication for access. You will need a provided `public key` to configure your destination. It will look roughly like this: ```text title="Public key format" icon="file-lines" theme={null} ssh-key some-comment ``` Log in to the SFTP server and complete the steps below. 1. Create group `sftpwriter`: ```shell title="Create group" icon="terminal" theme={null} sudo groupadd sftpwriter ``` 2. Create user `sftpwriter`: ```shell title="Create user" icon="terminal" theme={null} sudo useradd -m -g sftpwriter sftpwriter ``` 3. Switch to the `sftpwriter` user: ```shell title="Switch user" icon="terminal" theme={null} sudo su - sftpwriter ``` 4. Create the `.ssh` directory: ```shell title="Create .ssh directory" icon="terminal" theme={null} mkdir ~/.ssh ``` 5. Set permissions: ```shell title="Set directory permissions" icon="terminal" theme={null} chmod 700 ~/.ssh ``` 6. Navigate to the `.ssh` directory: ```shell title="Navigate to .ssh" icon="terminal" theme={null} cd ~/.ssh ``` 7. Create the `authorized_keys` file: ```shell title="Create authorized_keys file" icon="terminal" theme={null} touch authorized_keys ``` 8. Set permissions: ```shell title="Set file permissions" icon="terminal" theme={null} chmod 600 authorized_keys ``` 9. Add the public key to the `authorized_keys` file. *The key, including the "ssh-key" and comment, should be all on one line in the file, without linebreaks.* ```shell title="Add public key" icon="terminal" theme={null} echo "ssh-key sftpwriter-public-key" > authorized_keys ``` Use the following details to complete the connection setup: **host name**, **folder name**, **username**, **port** and preferred **delimiter character**. **Write permissions at the SFTP root are required** In addition to write access within your configured ``, this destination writes per-transfer manifest files under a `_manifests/` directory created at the root of the SFTP home/path. Ensure the SFTP user can create and write to `_manifests` at that root (even if your data lands under a subfolder). Manifests allow downstream systems to detect when a transfer is complete. See the FAQ below for how these files are organized. **Connection Test File Cleanup (Optional)** By default, a connection test is performed against the SFTP destination during initial configuration. This test writes temporary artifacts under the prefix `_test_connection/` (located at either the SFTP root, or inside the custom folder prefix configured on the destination, if present). If the SFTP user does not have permission to delete files, the connection test will still succeed, but these test files will remain on your SFTP server. To avoid clutter, you can optionally configure a custom periodic script (such as a cron job running on your SFTP host) to delete files/directories matching the `_test_connection/` prefix (or `/_test_connection/` if a custom folder was configured), or delete them manually. **Optional: PGP encryption** SFTP files are encrypted in transit, by virtue of the SFTP protocol. We offer an optional, additional layer of encryption at rest for SFTP files using PGP encryption. To enable PGP encryption, you will need to generate your own PGP public/private key pair, and provide the public key on a per-destination basis (one for each destination) when configuring each PGP-enabled destination. The public key must be provided in ASCII armored format, beginning with the header line `-----BEGIN PGP PUBLIC KEY BLOCK-----`, and ending with the tail line `-----END PGP PUBLIC KEY BLOCK-----`. For security reasons, only RSA and ECC keys are supported. RSA keys must have a key size of 2,048 bits or more. If PGP encryption is enabled, both the manifest files and landed data will be encrypted in the PGP binary format. They will have an additional `.pgp` file extension appended to their filename. For example, encrypted CSV files will have filenames like `your_data.csv.pgp`. It is not possible to enable or disable the PGP encryption settings of an existing destination. You must create a new destination from scratch with the new PGP configuration. ## Permissions checklist * SFTP user created with SSH key-based authentication. * Provided public key added to `~/.ssh/authorized_keys` for the SFTP user. * SFTP user has write access to the configured folder. * SFTP user can create and write to `_manifests/` at the SFTP home/path root. * Firewall or network perimeter allows the service's egress IP to connect on port 22. * *(If using PGP encryption)* PGP public key in ASCII armored format is ready to provide during destination configuration. RSA keys must be 2,048 bits or larger. ## Frequently asked questions The data will be loaded with the configured file format (`Parquet`, `CSV`, or `JSON`/`JSONL`) in a predictable folder structure that can be easily parsed by downstream systems. Use SSH key-based authentication for a dedicated, least-privileged SFTP user. Restrict access to only the required directories (e.g., chroot), and allowlist the service's static egress IP at your network perimeter. `Parquet` (default/recommended), `CSV`, and `JSON`/`JSONL`. Each transfer writes a manifest JSON file per model under `_manifests/` at the root. Files follow the pattern: `_manifests//dt=/manifest_{transfer_id}.json`. Use these manifests to trigger downstream processing. File-based destinations are append-oriented. The change-detection process uses a lookback window to prevent missed records, which can create duplicates across adjacent transfers. Downstream pipelines can deduplicate by primary key prioritizing rows in the most recent transfer window. We do not support providing your own public key for security reasons. The private key is securely generated and stored in our system and is never shared externally. Yes, you can optionally provide your own PGP public key for us to encrypt the SFTP files with, on a per-destination basis (one for each destination). The public key must be provided in ASCII armored format. No, the PGP key is not an alternative to the mandatory SSH/SFTP key. The PGP key is optional, while the SFTP key is mandatory. The two kinds of keys serve functionally distinct purposes. The SSH/SFTP key is used for authentication and encryption in transit. On the other hand, the PGP key is not used for authentication, and is instead used to add another layer of encryption of files in transit, and importantly, to ensure that the files remain encrypted at rest in the SFTP destination. The PGP encryption is applied within our system, after the data is pulled from the source. The data remains PGP-encrypted in transit between our system and the SFTP server, and stays PGP-encrypted at rest in the SFTP destination. Files are encrypted in the binary format only. If PGP encryption is enabled, both the manifest files and landed data will be encrypted. RSA key sizes must be 2,048 bits or more. # Snowflake Source: https://docs.prequel.co/export/destinations/snowflake Configuring your Snowflake destination. ## Prerequisites * Locate your Public Key generated on your behalf. The Public Key will be a long string of text, loosely resembling the format: `'MIIBI......Xrw2nwIDAQAB'` * In order to complete the following setup steps, you or a Snowflake admin on your team must have the securityadmin and sysadmin roles. (To check your account for these roles, run `SHOW GRANTS TO USER ;` and review the `role` column.) * If your Snowflake data warehouse is using Snowflake Access Policies, you will need to have our static IP available to complete Step 2. **Recommendation: Key-pair authentication with service user** Snowflake is deprecating single-factor passwords and will disallow passwords for service users (TYPE=SERVICE) by October 2026. For that reason, we strongly recommend configuring the transfer user as a service user with key-pair authentication. 1. Review and make any changes to the following setup script. ```sql title="Setup script (key-pair auth)" icon="database" expandable theme={null} begin; -- create variables for user / role / warehouse / database set user_name = 'TRANSFER_USER'; -- all letters must be uppercase set role_name = 'TRANSFER_ROLE'; -- all letters must be uppercase set warehouse_name = 'TRANSFER_WAREHOUSE'; -- all letters must be uppercase set database_name = 'TRANSFER_DATABASE'; -- all letters must be uppercase -- change role to securityadmin for user / role steps use role securityadmin; -- create role for data transfer service create role if not exists identifier($role_name); grant role identifier($role_name) to role SYSADMIN; -- establish SYSADMIN as the parent of the new role. Note: this does not grant the access privileges of SYSADMIN to the new role. -- create a user for data transfer service create user if not exists identifier($user_name) RSA_PUBLIC_KEY='MIIBIjANBgkqh...'; -- replace with the complete public key as required in the prerequisite -- set default role and warehouse to new user alter user identifier($user_name) SET default_role = $role_name; alter user identifier($user_name) SET default_warehouse = $warehouse_name; alter user identifier($user_name) SET type = service; grant role identifier($role_name) to user identifier($user_name); -- change role to sysadmin for warehouse / database steps use role sysadmin; -- create a warehouse for data transfer service create warehouse if not exists identifier($warehouse_name) warehouse_size = xsmall warehouse_type = standard auto_suspend = 60 auto_resume = true initially_suspended = true; -- create database for data transfer service create database if not exists identifier($database_name); -- grant service role access to warehouse grant USAGE on warehouse identifier($warehouse_name) to role identifier($role_name); -- grant service access to database grant CREATE SCHEMA, MONITOR, USAGE on database identifier($database_name) to role identifier($role_name); commit; ``` This script creates the new user using username & password authentication. The `create user` block uses `password` instead of `RSA_PUBLIC_KEY`. ```sql title="Setup script (password auth)" icon="database" expandable theme={null} begin; -- create variables for user / role / warehouse / database set user_name = 'TRANSFER_USER'; -- all letters must be uppercase set role_name = 'TRANSFER_ROLE'; -- all letters must be uppercase set warehouse_name = 'TRANSFER_WAREHOUSE'; -- all letters must be uppercase set database_name = 'TRANSFER_DATABASE'; -- all letters must be uppercase -- change role to securityadmin for user / role steps use role securityadmin; -- create role for data transfer service create role if not exists identifier($role_name); grant role identifier($role_name) to role SYSADMIN; -- establish SYSADMIN as the parent of the new role. Note: this does not grant the access privileges of SYSADMIN to the new role. -- create a user for data transfer service create user if not exists identifier($user_name) password = 'some_password'; -- set default role and warehouse to new user alter user identifier($user_name) SET default_role = $role_name; alter user identifier($user_name) SET default_warehouse = $warehouse_name; alter user identifier($user_name) SET type = service; grant role identifier($role_name) to user identifier($user_name); -- change role to sysadmin for warehouse / database steps use role sysadmin; -- create a warehouse for data transfer service create warehouse if not exists identifier($warehouse_name) warehouse_size = xsmall warehouse_type = standard auto_suspend = 60 auto_resume = true initially_suspended = true; -- create database for data transfer service create database if not exists identifier($database_name); -- grant service role access to warehouse grant USAGE on warehouse identifier($warehouse_name) to role identifier($role_name); -- grant service access to database grant CREATE SCHEMA, MONITOR, USAGE on database identifier($database_name) to role identifier($role_name); commit; ``` **Using an existing `schema`** By default, a new schema (with a name you provide) will be created in the target Snowflake database upon the initial connection. If instead you create the `schema` ahead of time, you may remove the `CREATE SCHEMA` permission, and instead `grant ALL PRIVILEGES` on the target `schema` for the designated `role`. The script below can be used to complete this step: ```sql title="Grant on existing schema" icon="database" expandable theme={null} set role_name = 'TRANSFER_ROLE'; set database_name = 'TRANSFER_DATABASE'; set schema_name = 'PRECREATED_SCHEMA'; use database identifier($database_name); grant ALL PRIVILEGES on schema identifier($schema_name) to role identifier($role_name); ``` **Using an existing `warehouse` or `database`** By default, this script creates a new warehouse and a new database. If you'd prefer to use an existing warehouse/database, change the `warehouse_name` variable from `TRANSFER_WAREHOUSE` to the name of the warehouse to be shared/`database_name` variable from `TRANSFER_DATABASE` to the name of the database to be shared. 2. In the Snowflake interface, select the dropdown next to the "Run" button, and click **Run All**. This will run every query in the script at once. If successful, you will see `Statement executed successfully` in the query results. If your Snowflake data warehouse is using Snowflake Access Policies, a new policy must be added to allow our static IP to write to the warehouse. 1. Review current network policies to check for existing IP safelists. ```sql title="Show network policies" icon="database" theme={null} SHOW NETWORK POLICIES; ``` 2. If there is no existing Snowflake Network Policies (the `SHOW` query returns no results), you can skip to Step 3. 3. If there is an existing Snowflake Network Policy, you must alter the existing policy or create a new one to safelist our static IP address. Use the `CREATE NETWORK POLICY` command to specify the IP addresses that can access your Snowflake warehouse. ```sql title="Create network policy" icon="database" theme={null} CREATE NETWORK POLICY ALLOWED_IP_LIST = (''); ``` **Network allowlisting** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. **Creating your first network policy** If you have no existing network policies and you create your first as part of this step, all other IPs outside of the `ALLOWED_IP_LIST` will be blocked. Snowflake does not allow setting a network policy that blocks your current IP address. (An error message results while trying to create a network policy that blocks the current IP address.) But be careful when setting your first network policy. Use the following details to complete the connection setup: **host name**, **database name**, your chosen **schema name**, **username**, and **password**. ## Permissions checklist * Role grants: * `USAGE` on the target warehouse * If the destination schema will be created by the service: * `USAGE` and `CREATE SCHEMA` on the target database (the setup script also includes `MONITOR`) * If using a pre-created schema: * `USAGE` on the target database * `ALL PRIVILEGES` on the target schema * User defaults set (optional but recommended): `DEFAULT_ROLE`, `DEFAULT_WAREHOUSE` * If using key auth: user has the PKCS#8 `RSA_PUBLIC_KEY` set * If network policies are enforced: our egress IP is allowlisted **Avoid `FUTURE GRANTS` on ownership to non-transfer Snowflake roles** The transfer role must retain ownership of the tables and internal stages it creates. Please ensure `FUTURE OWNERSHIP` on tables in the destination schema is not granted to other roles in Snowflake. ## FAQ We recommend key-based authentication. You register a public key on a Snowflake user and we authenticate using the corresponding private key, so no password is shared or stored. You can also enforce Snowflake Network Policies to allowlist our egress IP. Minimum grants: * `USAGE` on the warehouse * If the destination schema will be created by the service: `USAGE` and `CREATE SCHEMA` on the database * If using a pre-created schema: `USAGE` on the database and `ALL PRIVILEGES` on the schema Yes. Grant `USAGE` on that warehouse to the transfer role. You may also size the warehouse to control performance/cost. If you have multiple Snowflake destinations, you can use the same transfer role for up to 2 destinations. You will be issued a unique public key for each destination and can use the `RSA_PUBLIC_KEY` and `RSA_PUBLIC_KEY_2` to store up to two public keys. # Snowflake destination FAQs Source: https://docs.prequel.co/export/destinations/snowflake-faq Frequently asked questions when configuring and managing a Snowflake destination. See the following deep dive on the upcoming Snowflake authentication updates for your customers including timeline and migration steps: [Preparing for Snowflake Password Deprecation](/export/destinations/migrating-snowflake-authentication-to-key-pair). Continue reading below for additional information on managing this migration successfully alongside your customers. You can stage and promote a public key directly within the Admin UI or via the Prequel API for a new or existing destination: 1. **Admin UI** 1. When creating a new or editing an existing destination, you will see an option to switch Auth method from **User/Password** to **Key Auth**. **Prequel will automatically generate a public key for you.** 1. **Note:** You will need to first select a valid Recipient before a public key can be generated, as the key is staged to a Recipient before being promoted to a destination. 2. **Share this public key with your customer** so they can add it to their Snowflake user. See the main Snowflake destination documentation for your customer's next steps: [Snowflake](/export/destinations/snowflake). 1. **Note:** Newly generated public keys will expire after 30 days if not successfully promoted as described below. 3. Once your customer confirms the key has been added to their Snowflake environment, you can return to the Prequel Admin UI to complete creating the new destination or editing the existing one. 1. Note: Prequel will input the same public key into the destination form after one has been generated and staged for a given recipient. In other words, you can return to editing or creating a destination without saving and the public key shared with your customer will remain valid. 4. Once the new or existing destination is successfully saved, this promotes the public key within Prequel. 2. **Prequel API** 1. Use the [Create SSH Key](/export/api-reference/overview) endpoint to generate a key pair. The `public_key` that you will need to provide to your customer for their Snowflake user will be returned in the API response. 2. Make a POST or PATCH request to the [Create Destination](/export/api-reference/overview) or [Update Destination](/export/api-reference/overview) endpoints, providing the public key to promote it within Prequel. 1. Follow the steps above to stage a new public key from either the Admin UI or Prequel API. 2. Your customer should add this new key to their existing Snowflake user as `RSA_PUBLIC_KEY_2`, which allows both the old and new keys to be active in Snowflake simultaneously. 3. Once the new key is added in Snowflake, return to the Prequel Admin UI (or use the API) to **update the existing destination, selecting the newly generated public key.** 4. **Saving the destination in Prequel promotes the new key**, causing Prequel to immediately begin authenticating with Snowflake using the private key corresponding to `RSA_PUBLIC_KEY_2`. Since both keys are active in Snowflake, this ensures a seamless transition with no downtime. Snowflake keys are not rotated automatically. Prequel enables you to manage key rotation on behalf of your customers. No, Prequel does not currently support customers bringing their own public key for Snowflake authentication. For robust security and to ensure the private key never leaves our secure premises, Prequel generates and securely manages the entire key pair. No, it is not possible to reuse a Snowflake key across multiple destinations. Each destination requires a distinct key. Your customer can use `RSA_PUBLIC_KEY_2` on the same Snowflake user. # SQL Server Source: https://docs.prequel.co/export/destinations/sql-server Configuring your SQL Server destination. ## Prerequisites * If your SQL Server database is protected by security groups or other firewall settings, you will need our static IP available to complete Step 1. * Confirm that your SQL Server database is configured to allow TCP/IP connections. **Network allowlisting** Cloud Hosted (US): `35.192.85.117/32` Cloud Hosted (EU): `104.199.49.149/32` If private-cloud or self-hosted, contact support for the static egress IP. Create a rule in a security group or firewall settings to whitelist: * incoming connections to your host and port (usually `1433`) from the static IP. * outgoing connections from ports `1024` to `65535` to the static IP. Create a database user to perform the writing of the source data. 1. Open a connection to your SQL Server database. 2. Create a user for the data transfer by executing the following SQL command. The `` should be the target destination database. ```sql title="Create writer user" icon="database" expandable theme={null} USE ; CREATE LOGIN WITH PASSWORD = ''; CREATE USER FOR LOGIN ; ``` 3. Grant user `CREATE TABLE` privileges on the database. ```sql title="Grant CREATE TABLE" icon="database" theme={null} GRANT CREATE TABLE TO ; ``` **Understanding the `CREATE TABLE` permission in SQL Server** The `CREATE TABLE` permission is a database level permission that allows for the creation of new tables in a given database. The user must also have the `ALTER` permission granted on a given schema in order to create new tables in that schema (see the next step for details). 4. Grant user `CREATE SCHEMA` privileges on the database *if the schema does not exist*. ```sql title="Grant CREATE SCHEMA" icon="database" theme={null} GRANT CREATE SCHEMA TO ; ``` **If the `SCHEMA` already exists** By default, the service creates a new schema based on the destination configuration. If you prefer to create the schema yourself before connecting the destination, you must ensure that the writer user has the proper permissions on the schema, using: ```sql title="Grant schema privileges" icon="database" theme={null} GRANT SELECT, INSERT, UPDATE, DELETE, ALTER ON SCHEMA :: TO ; ``` If the `SCHEMA` already exists, the user does not need the `GRANT CREATE SCHEMA` permission. Use the following details to complete the connection setup: **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password**. **Credential character limitations** For user credentials containing special characters, please avoid using the following characters: `@`, `[`, `]`, `/`, `?`, `#`, `"`, `\\`, `+`, space, `&`, `:`, `;`, `%`, `=` as these characters can break connection string parsing. ## Permissions checklist * Network: * Inbound rule allows TCP `1433` from the static egress IP * Outbound rule allows ephemeral ports `1024-65535` to the static egress IP * SQL Server: * `CREATE TABLE` on the target database * If schema is created by the service: `CREATE SCHEMA` on the database * If schema is pre-created: `SELECT, INSERT, UPDATE, DELETE, ALTER` on the target schema * TCP/IP connections are enabled * Optional: * If connecting via SSH tunnel: bastion host allows inbound SSH from static egress IP, SSH user created with service public key, bastion IP granted access to database port ## FAQ The connection uses a dedicated, least-privileged SQL login scoped to the destination database and schema. Network access can be restricted to the static egress IP. For databases not accessible from the public internet, SSH tunneling through a bastion host is supported. Avoid these characters in usernames and passwords because they can break connection string parsing: `@`, `[`, `]`, `/`, `?`, `#`, `"`, `\\`, `+`, space, `&`, `:`, `;`, `%`, `=`. Generic on-premises SQL Server, Azure SQL Database, and Azure Synapse are supported. For Azure dedicated SQL pools, we recommend using the Azure Blob Storage destination type and loading from Azure Data Lake Storage Gen2. # AWS SSH tunneling Source: https://docs.prequel.co/export/destinations/ssh-tunneling Configure an AWS bastion server to accept an SSH connection from our service. Follow these steps to configure an AWS bastion server to accept an SSH connection from our service. ## Prerequisites * A public SSH key to add to the bastion machine to authenticate the SSH connection. *Ask your contact at our service if they have not already provided one to you.* If using SSH tunneling to connect to a destination database, we recommend using a standalone bastion server. This step will walk through provisioning the standalone bastion server. If using an existing server, skip to step 2. 1. Navigate to the EC2 Instances page, and click **Launch instances**. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-launch-instances.png "launch instances.png") 2. Select an image. We recommend a default Amazon Linux image, such as the *64-bit (x86) Amazon Linux 2 AMI* (the default option). ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-select-image.png "image.png") 3. Select a type. We recommend the **t3.nano**. *Because this bastion server will be used exclusively for SSH tunneling, Network Performance is the primary consideration. The t3.nano default configuration comes with Network Performance of up to 5 Gigabit, as well as minimal Memory (1 GiB), vCPUs (2) and Storage (8 GB SSD) for about \$5/mo depending on region and market rates.* ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-t3-nano.png "t3nano.png") ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-t3-nano-cost.png "t3nano cost.png") 4. Click **Review and launch**. Consider modifying the default security group according to your security policies to enable configuration (e.g., creating the SSH user in **Step 2**). Click **Launch**. We will further modify the default security group in **Step 2**. (The remaining default options are sufficient.) ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-select-launch.png "launch.png") ### Allow SSH port access In the AWS console, complete the steps below 1. Navigate to the **EC2 Instances** page, and click the **instance ID** of the server to be used as the bastion host. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-instance-id.png "ec2 instance id 2.png") 2. In the **Security** tab, click the link to one of the security groups. (E.g., the **default** security group). Make note of this security group, as you will need to allow inbound access to your destination DB as part of **Step 3**. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-security-tab.png "instance security tab.png") 3. In the **Inbound rules** tab, click "Edit inbound rules". ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-inbound-rules.png "ec2 edit inbound rules.png") 4. Click **Add rule**. Add an inbound rule of type **SSH** and whitelist our static IP. Optionally, add a description. For the correct IP, refer to the [IPs](/export/deployment/prequel-ips) section of the docs. \_ **Note**: you will not be able to SSH into this machine to configure the SSH user unless you also whitelist your machine's IP in the inbound rules. Consider temporarily adding that IP to facilitate the next step.\_ ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-whitelist-prequel-ip.png "ssh tunnel whitelist.png") ### Create the SSH user Login to the bastion server and complete the steps below. 1. Create group ``: ```shell title="Create group" icon="terminal" theme={null} sudo groupadd ``` 2. Create user ``: ```shell title="Create user" icon="terminal" theme={null} sudo useradd -m -g ``` 3. Switch to the SSH user: ```shell title="Switch to SSH user" icon="terminal" theme={null} sudo su - ``` 4. Create the `.ssh` directory: ```shell title="Create .ssh directory" icon="terminal" theme={null} mkdir ~/.ssh ``` 5. Set permissions: ```shell title="Set directory permissions" icon="terminal" theme={null} chmod 700 ~/.ssh ``` 6. Navigate to the `.ssh` directory: ```shell title="Navigate to .ssh directory" icon="terminal" theme={null} cd ~/.ssh ``` 7. Create the `authorized_keys` file: ```shell title="Create authorized_keys file" icon="terminal" theme={null} touch authorized_keys ``` 8. Set permissions: ```shell title="Set file permissions" icon="terminal" theme={null} chmod 600 authorized_keys ``` 9. Add the public key to the `authorized_keys` file. *The key, including the "ssh-key" and comment, should be all on one line in the file, without linebreaks.* ```shell title="Add public key" icon="terminal" theme={null} echo "" > authorized_keys ``` 1. Within the **Inbound rules** of the database security group, click **Add rule**. Add the security group noted in **Step 2**, and click **Save rules**. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-bastion-rule.png "Grant db access to bastion server.png") # Azure Blob Storage staging bucket Source: https://docs.prequel.co/export/destinations/staging-abs Configuring your ABS bucket for staging data to load into your destination. Some sources or destinations without built-in staging resources require a staging bucket to efficiently transfer data. ## Create storage account 1. In the Azure portal, navigate to the **Storage accounts** service and click **+ Create**. 2. In the "Basics" tab of the "Create a storage account" form, fill in the required details. 3. In the "Advanced" settings, under "Security" make sure **Enable storage account key access** is turned on. You may turn off (deselect) "Allow enabling public access on containers". Under "Data Lake Storage Gen2", select **Enable hierarchical namespace**. ![](https://storage.googleapis.com/prequel_docs/images/azure-settings-toggles.png "Enable hierarchical namespace and other settings") 4. In the "Networking" settings, you may limit "Network access" to select **Enable public access from all networks** to ensure your destination can utilize this resource. All other settings can use the default selections. 5. In the "Data protection" settings, you must turn off **Enable soft delete for blobs**, **Enable soft delete for containers**, and **Enable soft delete for file shares**. ![](https://storage.googleapis.com/prequel_docs/images/azure-turn-off-settings.png "Disable default soft delete settings") 6. Once the remaining options have been configured to your preference, click **Create**. ## Create bucket and access token 1. In the Azure portal, navigate to the **Storage accounts** service and click on the account that was created in the previous step. 2. In the navigation pane, under "Data storage", click **Containers**. Click **+ Container**, choose a name for the container, and click **Create**. 3. In the navigation pane, under "Security + networking", click **Shared access signature**. 4. In the "Allowed services" list, select **Blob** and **File**. In the "Allowed resource types" list, select **Container** and **Object**. In the "Allowed permissions" list, select **Read**, **Write**, **Delete**, **List**, **Add**, **Create**, and **Permanently Delete**. 5. Select a "Start and expiry date/time" based on your security posture, and click **Generate SAS and connection string**. 6. Make a note of the **SAS token** that is generated. ![](https://storage.googleapis.com/prequel_docs/images/azure-setup-details.png "Generate SAS token and connection string") ## You're done! Use this configured Azure Blob Storage staging bucket during the connection of your preferred data destination. # Google Cloud Storage staging bucket Source: https://docs.prequel.co/export/destinations/staging-gcs Configuring your GCS bucket for staging data to load into your destination. Some sources or destinations without built-in staging resources require a staging bucket to efficiently transfer data. ## Create a service account 1. In the GCP console, navigate to the **IAM & Admin** menu. 2. Click into the **Service Accounts** tab. 3. Click **Create service account** at the top of the menu. 4. In the first step, name the service account that will be used to transfer data into Cloud Storage and click **Create and Continue**. You may continue through the following steps without assigning any roles, and in the final step, click **Done**. ## Create staging bucket 1. Navigate to the **Cloud Storage** page. 2. Click **Create**. 3. Enter a **bucket name**, choose a **region**. 4. After choosing your preferences for the remaining steps, click **Create**. *(If presented with a warning, you may enforce public access prevention)* 5. On the **Bucket details** page for the bucket you created, select the **Permissions** tab, and click **Grant access**. 6. Grant access to the principal (Service Account) you created in **Step 1**, and assign the Roles: **Storage Object Creator** and **Storage Object Viewer**. Click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-grant-staging-bucket-access.png) ## Generate HMAC key An HMAC key is a type of credential and can be associated with a service account or a user account to access Google Cloud Storage. 1. Navigate to the **Cloud Storage** page. 2. Click into the **Settings** tab on the left side menu. 3. In the **Interoperability** tab, click the **Create a key for a Service Account** button. 4. Select the **Service Account** created in **Step 1**, and click **Create key**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-staging-bucket-access-key.png) 5. Make a note of the **Access key** and **Secret**. ## You're done! Use this configured GCS staging bucket during the connection of your preferred data source or destination. Depending on the connection method, you may need to refer to the following field name mappings: * **Bucket Access ID**: your HMAC Access Key. * **Bucket Secret Key**: your HMAC Secret. # S3 staging bucket Source: https://docs.prequel.co/export/destinations/staging-s3 Configuring your AWS S3 bucket for staging data to load into your destination. Some destinations without built-in staging resources require a staging bucket to efficiently load data. ## Prerequisites By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` ## Create staging bucket 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "**ACLs disabled**" and **Block Public Access settings for this bucket** can be set to "**Block all public access**" as recommended by AWS. Make note of the Bucket name and AWS Region. 4. Click **Create bucket**. ## Create policy 1. Navigate to the **IAM** service page, click on the **Policies** navigation tab, and click **Create policy**. 2. Click the JSON tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen above. 1. **Note**: the first policy applies to `BUCKET_NAME` whereas the second policy applies only to the bucket's contents (`BUCKET_NAME/*`), an important distinction. ```json title="IAM policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` 3. Click through to the **Review** step, choose a **name** for the policy, for example, `transfer-service-policy` (this will be referenced in the next step), add a description, and click **Create policy**. ## Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. ## You're done! Use this configured S3 staging bucket during the connection of your preferred data source or destination. # Prequel error codes Source: https://docs.prequel.co/export/error-handling/error-codes Reference of Prequel error codes, their meaning, and how to resolve them. Every Prequel error maps to a stable code in the `PRQL-XXXX` format. Prequel matches the raw error returned by a source or destination against a library of known failure modes. For more information on why Prequel serves a categorized error code instead of raw traces from the offending driver/vendor, please see [Error handling](/export/error-handling/error-handling). Each code also carries a `blame` value, so you can tell whether you, your customer, or Prequel needs to act before anyone starts investigating. An error that Prequel has not yet encountered or classified returns `PRQL-ERMS` and preserves the original error in the `trace` field. Please reach out to the Prequel team if you encounter such an error so it can be addressed and added promptly. ## Code reference | Code | Title | Message | | :-------- | :---------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PRQL-RZBD | All Records Errored | None of the source provided rows were processed successfully. Please review the error records to determine the cause of the failure. | | PRQL-BNOD | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please check the s3:ListBucket permission. | | PRQL-DDAK | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please check for Create Database on Catalog permission. | | PRQL-FDEW | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. | | PRQL-RWCV | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please verify the glue:DeleteTable permission exists on the generated tables. | | PRQL-VDMC | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please check the s3:GetObject permission. | | PRQL-XZWE | Athena Cannot Find Requested Iceberg Entity | Athena has detected a corrupted Iceberg table. Full refresh to resolve. | | PRQL-ONNN | Athena Connection Refused | The Athena connection was refused. Please check that the provided Athena access details are correct. | | PRQL-QDPO | Athena DDL Execution Timeout Error | The Athena query exceeded the engine's DDL query timeout. If this error persists, please contact support. | | PRQL-XJWM | Athena Data Catalog Timeout Error | The Athena data catalog connection timed out. If this error persists, please contact support. | | PRQL-AFOK | Athena Data Processing Failed | The Athena engine encountered a ConcurrentModificationException while executing a query. | | PRQL-AWRI | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process Iceberg formatted data. | | PRQL-BKDL | Athena Data Processing Failed | The Athena query engine encountered an error. A value found in the underlying source data is incompatible with the type double defined in the Athena table schema. Please review the source's data type and either cast the type or update your Prequel data model accordingly. | | PRQL-BSAX | Athena Data Processing Failed | The Athena engine encountered an error related to the timestamp precision on some timestamp data. | | PRQL-CFGX | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process data in hive mode. | | PRQL-FLUU | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process Iceberg formatted data. | | PRQL-GVZM | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process Hive formatted data. | | PRQL-CERZ | Athena Federated Read Error | Athena encountered an exception reading from a federated data source. Please refer to Athena logs for more information. | | PRQL-KXAL | Athena Glue Data Catalog Rate Limit Exceeded | AWS throttled Glue Data Catalog API requests for this Athena connection. This is usually transient; please retry, and reduce concurrent Athena operations if it persists. | | PRQL-DJDS | Athena Glue Metastore Operation Timeout | The Glue Data Catalog timed out fetching table metadata for the Athena source. Please review Glue API usage from concurrent operations, reduce contention if possible, and if necessary, request an increase in service quota. | | PRQL-JNZR | Athena IAM Role Token Audience Mismatch | AWS rejected the federated identity token because its audience did not match what the IAM role expects. Please verify the oaud condition in the role's trust policy matches the provided value. | | PRQL-YOTO | Athena Iceberg Partitions Exceeded | The Athena engine exceeded the maximum number of open partition writers. Please contact support for more information. | | PRQL-IVCA | Athena Invalid Cast Argument | One or more column was unable to be cast to the data type from the corresponding model. Please ensure that are source column data types are compatible with model data types and try again. | | PRQL-MSQX | Athena Invalid Date Value | Athena could not cast a data value to type DATE. Check the source data to ensure dates are in the format YYYY-MM-DD. | | PRQL-TKVM | Athena Invalid Timestamp Value | Athena could not cast a data value to type TIMESTAMP. Check the source data to ensure the data can be properly cast into timestamp or determine if it requires additional handling. | | PRQL-LFDR | Athena Lake Formation Permissions Error | The connection encountered one or more Lake Formation permissions errors. Please ensure any Lake Formation policy contains sufficient permissions and try again. Contact support if the issue persists. | | PRQL-LRCT | Athena Lambda Timeout | Athena encountered an exception reading from a federated data source. A lambda function timed out. | | PRQL-CKSC | Athena Missing Column | The Athena query engine encountered an error. A column name was not recognized. | | PRQL-KZWP | Athena NOT NULL Value Constraint Failed | A NULL value was found in a column with non-null requirements. Please check the model's columns designated as is\_primary\_key and is\_last\_modified for any null values in your source. | | PRQL-XIEA | Athena Query Timeout | The Athena query execution timed out. If this error persists, please contact support. | | PRQL-NXQP | Athena Read Error | The Athena engine encountered a problem while attempting to read the source view. Please confirm the view is not stale. | | PRQL-FJOD | Athena Runtime Encountered Error | The provided AWS role does not have access to the Athena Workgroup. If the Athena Workgroup does not exist, the provided role does not have permission to create the workgroup. | | PRQL-TJCZ | Athena Runtime Encountered Error | The Athena connection encountered a memory limit. | | PRQL-VKEV | Athena Runtime Encountered Error | The Athena catalog encountered an internal server error. If this error persists, please contact support. | | PRQL-WJTE | Athena Runtime Encountered Error | The Athena connection encountered an error. | | PRQL-WRFA | Athena Runtime Encountered Error | The Athena engine encountered an error. | | PRQL-YTNS | Athena Runtime Encountered Error | The Athena connection encountered a resource limit. | | PRQL-ASFB | Athena S3 Access Denied | The provided credentials were denied access to the S3 source bucket. Please check that the IAM role has s3:GetObject, s3:ListBucket, and s3:GetBucketLocation granted on the source bucket and its contents. | | PRQL-CMYT | Athena Schema Does Not Exist Error | The Athena schema could not be found. Please check that the schema exists and that the provided connection principal has been granted sufficient permissions. | | PRQL-HDXP | Athena Server Encountered Generic Error | The Athena connection encountered a generic internal error. If this error persists, please contact support. | | PRQL-KLBO | Athena Server Encountered Internal Error | The Athena connection encountered an internal error. If this error persists, please contact support. | | PRQL-GAIX | Athena Table Missing | The Athena query engine encountered an error. The table name was not recognized. | | PRQL-MZXD | Athena Table Not Found | The Athena engine could not find the table to read from or merge into. Please verify that the table exists and the connection principal has been granted sufficient permissions. | | PRQL-XYYX | Athena Type Mismatch | Athena attempted to use an operator on values of differing types. Please check the source query for type mismatches. | | PRQL-WHHO | Athena Unable To Assume Role | Athena was unable to assume the specified role. Please verify the role ARN. | | PRQL-VMEX | Athena Unsupported Timestamp Precision Error | Athena source data of type "timestamp(6)" is unsupported. Please expose the column as type "timestamp(3)". | | PRQL-QTSM | Athena Unsupported Timestamp Type Error | Athena source data of type "timestamp with time zone" is unsupported. Please expose the column as type "timestamp". | | PRQL-MDSR | Azure Authorization Expired | The Azure server returned an error indicating the access token is expired. Please check that the provided shared access key is valid and has been granted sufficient permissions. | | PRQL-BFKH | Azure Blob Snapshots Prevented Delta Table Cleanup | A file in the destination bucket could not be deleted because it has blob snapshots. Please first delete all snapshots of blobs under the table prefix, and ensure soft deletes are disabled on the storage account. Then, try again. | | PRQL-ZDJR | Azure Blob Storage Authentication Failed | Azure Blob Storage encountered an error. Please check that the provided connection principal has been granted sufficient permissions. | | PRQL-UXWZ | Azure Blob Storage Authorization Error | Azure Blob Storage returned an authorization error. Please verify your SAS token includes the required permissions on the target Container and has not expired. If network access is restricted on the storage account, ensure the service egress IP is in the allowed range. | | PRQL-GJWR | Azure Blob Storage Host Not Found | The Azure Blob Storage host could not be found. Please check that the provided host name is valid. | | PRQL-VMDR | Azure Blob Storage Invalid SAS Token | Azure Blob Storage could not authenticate the request. Please verify you have generated an SAS token (access keys are not supported), your SAS token has not expired, and was copied correctly, then try again. | | PRQL-YZQZ | Azure Databricks Storage Request Not Authorized | The Azure Databricks storage request failed authorization. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Please verify your Azure storage credentials or firewall exception settings. | | PRQL-JRKJ | BigQuery Authentication Failed | The BigQuery connection was refused. Please check that that the provided connection principal has been granted sufficient access. | | PRQL-PVER | BigQuery Authentication Failed | The BigQuery connection was refused. Please check that that the provided connection principal has been granted sufficient access. | | PRQL-SPJT | BigQuery Authentication Failed | The BigQuery server rejected the connection. Please check that the provided authentication principal and credentials are valid. | | PRQL-AVSM | BigQuery Avro File Schema Mismatch | An Avro file backing this BigQuery table is missing a column the table schema requires. Please check the pipeline writing the table's underlying files, or add a default value for the column. | | PRQL-UJSZ | BigQuery Column Already Exists | The BigQuery engine encountered an error. The target column name already exists. | | PRQL-NIKQ | BigQuery Column Does Not Exist | The BigQuery engine encountered an error. The target column name is unrecognized. | | PRQL-BJRB | BigQuery Column Rejected Null Value | A column in the destination table is marked REQUIRED but received a null value. This can occur when a column present in the destination table is missing from the write. Please contact support for more information. | | PRQL-VKGX | BigQuery Column Type Mismatch | One or more column's data type does not match the existing destination column. Please verify all destination table column types match the types defined in the corresponding model. | | PRQL-UCFB | BigQuery Connection Refused | The provided Google Cloud project has not enabled BigQuery. | | PRQL-CQEX | BigQuery Custom Query Quota Exceeded | A custom BigQuery query usage quota set by your administrator was exceeded. Please raise the quota or wait for it to reset. | | PRQL-FGWV | BigQuery Data Processing Failed | The BigQuery data transfer encountered an error during data transfer. The BigQuery target table may have duplicate data preventing successful updates. | | PRQL-JEBJ | BigQuery Dataset Not Found | The specified BigQuery dataset was not found in the configured project and region. Please verify the dataset name and region provided and try again. | | PRQL-TIRE | BigQuery Disallowed Bucket Retention Policy | Bucket and object retention policies are disallowed on BigQuery staging policies. Please remove the policy or use a bucket without a retention policy. | | PRQL-ZVKN | BigQuery GCS Authentication Error | The service account does not have sufficient access to the GCS bucket. Please review the list bucket permissions provided via IAM roles. | | PRQL-BRLC | BigQuery Invalid Bucket Region | The provided GCS bucket region is invalid. Please ensure the specified region field is exactly one of the following valid GCS bucket location names: [https://docs.cloud.google.com/storage/docs/locations](https://docs.cloud.google.com/storage/docs/locations). | | PRQL-WMRB | BigQuery Rate Limit Exceeded | One or more queries has exceeded a BigQuery rate limit. Please contact support for more information. | | PRQL-PWQD | BigQuery Runtime Encountered Error | The BigQuery connection encountered an error. | | PRQL-GKCB | BigQuery Server Encountered Error | The BigQuery API was temporarily unavailable (HTTP 503). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Cloud status page to confirm service health. | | PRQL-FGCV | BigQuery Service Account Disabled | The BigQuery connection was refused because the configured service account has been disabled. Please re-enable the service account in GCP IAM & Admin, or create a new service account and update configuration details. Then, try again. | | PRQL-AEVP | BigQuery Table Not Accessible | The user does not have permission to query the specified BigQuery Table, or it was not found in the configured region and dataset. Please review the user's permissions on the table and dataset, and the region and dataset names provided. | | PRQL-MJOZ | ClickHouse Connection Refused | The provided ClickHouse host was not found. Please check that the provided host name is valid and accessible. | | PRQL-OEVX | ClickHouse Connection Refused | The provided ClickHouse host was not found. Please check that the provided host name is valid and accessible. | | PRQL-UFTZ | ClickHouse Connection Refused | The provided ClickHouse host refused the connection. Please check that the provided host name is valid and accessible. | | PRQL-WKVZ | ClickHouse Connection Timed Out | The connection to the ClickHouse host timed out. Please check that the host and port are reachable and that the static IP address (or addresses) are allowlisted in your security group/firewall. | | PRQL-CTBL | ClickHouse Insufficient Create Table Permissions | The ClickHouse user does not have permission to create tables. Please ensure both CREATE TABLE and CREATE TEMPORARY TABLE are granted to the ClickHouse user on the target database and try again. | | PRQL-BKFC | ClickHouse Insufficient Read Permissions | The ClickHouse user does not have permission to read from the data tables and/or schema. Please ensure to grant SELECT on the database, tables for sync, and information\_schema and try again. | | PRQL-DSAJ | ClickHouse Insufficient S3 Staging Bucket List Permissions | The ClickHouse instance was denied access to the S3 staging bucket while checking whether a staged file exists before writing. Please check that you have provided the s3:ListBucket permission on the staging bucket itself (without any key prefix added), and try again. | | PRQL-WKRF | ClickHouse Insufficient S3 Write Permissions | The ClickHouse user cannot write to the S3 staging bucket. Please grant the S3 write permission to the ClickHouse user with GRANT S3 ON *.* to \@'%' (or GRANT WRITE ON S3 if your instance uses granular enable\_read\_write\_grants) and try again. | | PRQL-BNPN | ClickHouse Insufficient View Permissions | The ClickHouse user does not have permissions to create and/or drop views generated during transfers. Please grant CREATE VIEW and/or DROP VIEW to the ClickHouse user on the target database and try again. | | PRQL-SKFD | ClickHouse Invalid Connection Parameter | The provided ClickHouse credentials contain an invalid character. Please check that the username and password do not contain special characters such as @, /, or ?. | | PRQL-UQKO | ClickHouse Memory Limit Exceeded | The ClickHouse instance ran out of memory while exporting data. Please scale up the service's maximum memory limit and try again. | | PRQL-XCNS | ClickHouse Rate Limited | The Clickhouse server encountered a resource quota limit for simultaneous queries. | | PRQL-ZXEC | ClickHouse TLS Certificate Expired | The ClickHouse endpoint presented an expired TLS certificate. Please renew the certificate on the server or proxy terminating TLS and try again. | | PRQL-VCXL | Clickhouse Invalid Truncate Permissions | The Clickhouse engine encountered an error. Please check that the provided user has been granted sufficient permissions to truncate tables. | | PRQL-NJEW | Data Serialization Error | One or more rows from the source cannot be parsed. This may be caused by a row exceeding byte size limits, invalid/non-supported encoded characters, unquoted values, or column errors. Please contact support for more information. | | PRQL-BKRG | Databricks Authorization Failed | The provided service principal is not authorized to use the SQL endpoint. Please confirm that the service principal is listed as an authorized user with Can Use within the SQL Warehouse permissions, and try again. | | PRQL-AXJS | Databricks Catalog Not Found | The specified Databricks catalog was not found. Please check the catalog name in the connection configuration exists in the target workspace and try again. | | PRQL-CJFX | Databricks Catalog Permission Denied | The Databricks connection was refused due to insufficient catalog permissions. Please check that the provided connection principal has been granted USE CATALOG permission on both the target catalog and Databricks system catalog, then try again. | | PRQL-CSVN | Databricks Concurrent Write Conflict | The Databricks server encountered a concurrent update to the target table. Please ensure no other INSERT, DELETE, UPDATE, or MERGE queries are running. | | PRQL-LADF | Databricks Connection Closed Prematurely | The Databricks connection closed before the statement finished executing. Check the Databricks status page and confirm the target SQL warehouse is running and reachable. | | PRQL-MKEW | Databricks Connection Temporarily Rejected | The Databricks connection was temporarily rejected because the service is under maintenance. | | PRQL-ZQAC | Databricks Delta Table Metadata Missing From Storage | One or more Delta table transaction log files are missing from the underlying storage bucket. Please check bucket lifecycle policies to ensure no files are deleted, and contact support to learn more. | | PRQL-LRXO | Databricks Destination Table Missing Model Column | The destination table is missing a column the model expects. Please restore the column or run a full refresh to rebuild the table. | | PRQL-WBZK | Databricks Destination Table Ownership Required | The connection principal does not own the destination table. Please transfer ownership of the table to the connection principal, then retry. | | PRQL-VIDE | Databricks Duplicate Row Error | The Databricks data transfer encountered an error during data transfer. The Databricks target table or staged data may have duplicate data preventing successful updates. Please confirm there is no duplicate data in the data source, and contact support if the issue persists. | | PRQL-IPBL | Databricks IP Access List Restriction | The Databricks connection was blocked by an IP access list restriction at the Databricks account or workspace-level. Please ensure any configured IP access lists include the data sync service's static egress IP(s). | | PRQL-JMES | Databricks Identity Not Assigned To Workspace | The connection principal is not a member of the targeted Databricks workspace. Please confirm the user or service principal is assigned to that workspace and that the configured host points to the correct workspace, then try again. | | PRQL-XTZH | Databricks Insufficient Credits Available | The connection to Databricks could not be established as the Databricks account has insufficient credits. Please add a payment method or purchase additional credits. | | PRQL-OLKS | Databricks Internal Query Failure | Databricks reported an internal error while executing the query. Please check the Databricks status page and retry. | | PRQL-JZQZ | Databricks Invalid Access Token | The Databricks connection was refused due to an invalid access token. Please check that the access token (`dapi...`) is valid and the provided connection principal has been granted sufficient access. | | PRQL-OAUT | Databricks Invalid OAuth Credentials | The Databricks connection was refused because the OAuth client credentials were rejected. Please verify that the Client ID and Client Secret are correct and the credential has not expired, then please try again. | | PRQL-QOCK | Databricks Java IO Error Encountered | Databricks encountered an I/O error while executing the operation. Review the Databricks query/driver logs and the Databricks status page to identify the failure. | | PRQL-VIZF | Databricks Network Error Encountered | The connection to Databricks was dropped unexpectedly. Check the Databricks status page and confirm the target SQL warehouse is running and reachable. | | PRQL-FUOM | Databricks Operation Not Supported | The Databricks server encountered an error. Please check that the provided connection principal has been granted sufficient access. | | PRQL-QWEO | Databricks Query Timeout | The Databricks query connection timed out. If this error persists, please contact support. | | PRQL-CKSR | Databricks Rate Limit | The Databricks server encountered a rate limit. If this error persists, please contact support. | | PRQL-CYNR | Databricks Runtime Error Encountered | Databricks could not schedule the query and returned a temporary error (HTTP 503) asking to retry. This usually resolves on the next run. If the error persists, check the Databricks status page and the target SQL warehouse's health and load. | | PRQL-GMXD | Databricks Runtime Error Encountered | The Databricks server does not have Predictive Optimization enabled. | | PRQL-INXB | Databricks Serverless Warehouse Disabled | The Databricks warehouse cannot start because serverless warehouse compute is disabled in the global warehouse configuration. Please contact your Databricks administrator to enable compute for serverless warehouses, and try again. | | PRQL-YBLK | Databricks Service Unavailable | Databricks returned a 503 Service Unavailable response. This may be caused by Databricks SQL warehouse autoscaling, startup delays, warehouse load, or a transient Databricks service issue. Please check that both the target SQL warehouse and Databricks servers are healthy, and try again. | | PRQL-CMRT | Databricks Storage Authorization Failure | Databricks failed to execute one or more staging upload queries with PUT. If using Unity Catalog, please ensure the Write Volume permission has been provided to the Service Principal or user. If using Hive Metastore, please ensure write and delete permissions have been provided on the object storage location. | | PRQL-BUMS | Databricks Table Or View Not Found | A referenced table or view was not found. Please check that the target tables were not dropped or renamed. | | PRQL-PSZD | Databricks Warehouse Stopped | The Databricks transfer failed because the target warehouse was stopped. | | PRQL-YWEX | Databricks Workspace Rate Limit Exceeded | The Databricks workspace exceeded its API rate limit. Please retry later, reduce concurrent requests, and/or contact Databricks to increase request limits in the workspace. | | PRQL-DBSH | Databricks no such host | Connection to Databricks could not be established on failure to reach host. Please ensure that the provided hostname is accurate and the Databricks cluster is publicly accessible. | | PRQL-DLDQ | Delta Lake Duplicate Row Error | One or more duplicate rows have been detected in the source data. Please ensure that each row's primary key is unique/de-duplicated and retry. | | PRQL-WKXN | Delta Lake Engine Stopped Unexpectedly | The Delta Lake engine stopped unexpectedly during the transfer. Please retry the transfer. If this error persists, please contact support. | | PRQL-UFQE | Delta Lake Object Not Found | The Delta Lake engine could not find an object. Make sure the object exists and the writer has read access to the bucket. | | PRQL-GFIS | Delta Lake Runtime Error | The Delta Lake engine encountered an error. | | PRQL-CISE | Delta Lake Table Creation Failed | The delta lake table could not be created due to existing conflicting metadata in the bucket prefix. | | PRQL-GTRI | Delta Lake Transaction Log Commit Failed | The transaction log could not be committed to the destination bucket. This is most commonly caused by failing to disable soft delete defaults on the storage account. Please first check that both blob and container soft deletes are disabled on the storage account, then ensure no other engines write to the table concurrently before retrying. | | PRQL-BERH | Destination Table Not Found | The destination table was not found in the target schema. Please ensure the destination table exists at the configured schema and with the target table name. Contact support for more information. | | PRQL-DOOM | DuckDB Out of Memory | The transfer worker ran out of memory. If this error persists, please contact support. | | PRQL-PZCR | Federated Token Exchange Rejected | Google rejected the identity provider's token during the workload identity federation exchange. Verify the workload identity pool and provider settings, including the allowed audience, and that the identity provider issues JWT access tokens. | | PRQL-QGBC | GCS Bucket Not Found | A GCS bucket under the configured bucket name and region could not be found in the target GCP project. Please check that the name and region are set correctly, create the bucket if it does not already exist, then try again. | | PRQL-SLVK | GCS Missing Bucket Storage GET Permission | The GCS server encountered an error due to insufficient permissions. Please check that the provided connection principal has been granted 'storage.buckets.get'. | | PRQL-ZQZQ | GCS Missing Create Permission | The GCS server encountered an error due to insufficient permissions. Please check that the provided connection principal has been granted 'storage.objects.create'. | | PRQL-AYSD | GCS Missing Delete Permission | The GCS server encountered an error due to insufficient permissions. Please check that the provided connection principal has been granted 'storage.objects.delete'. | | PRQL-KJCH | GCS Server Encountered Error | Google Cloud Storage returned an internal server error (HTTP 500). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Cloud status page to confirm service health. | | PRQL-NXPY | GCS Server Encountered Error | Google Cloud Storage was temporarily unavailable (HTTP 503). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Cloud status page to confirm service health. | | PRQL-IXOL | Google Sheets Authentication Failed | The Google Sheets server refused the connection. | | PRQL-NEFP | Google Sheets Authentication Failed | The Google Sheets server refused the connection. | | PRQL-VDJX | Google Sheets Data Processing Failed | The Google Sheets server encountered a temporary error and the connection was dropped. If this error persists, please contact support. | | PRQL-LNWT | Google Sheets Host Not Found | The Google Sheets host could not be found. Please check that the provided host name is valid. | | PRQL-CELL | Google Sheets Overloading Error | A Google Sheet may not have more than 10,000,000 cells. If the error persists, please choose another destination type. | | PRQL-BOSR | Google Sheets Server Encountered Error | Google Sheets returned a bad gateway error (HTTP 502). This is typically a temporary issue and resolves automatically. | | PRQL-CMET | Google Sheets Server Encountered Error | Google Sheets was temporarily unavailable (HTTP 503). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Workspace status page to confirm service health. | | PRQL-GPDB | Google Sheets Server Encountered Error | Google Sheets returned an internal server error (HTTP 500). This is typically a temporary issue and resolves automatically. | | PRQL-YICW | Google Sheets Server Encountered Error | The Google Sheets server encountered a resource quota limit. | | PRQL-ICED | Iceberg Authorization Failed | A query failed due to insufficient database permissions. Please check that the provided connection details are correct and the principal has been granted sufficient permissions to the target catalog and object store. | | PRQL-FJVY | Identity Provider Rejected Client Credentials | The identity provider rejected the configured OAuth client credentials (invalid\_client). Verify the client ID and client secret and try again. | | PRQL-BVHD | Identity Provider Token Fetch Failed | Prequel could not fetch an OAuth token from the configured identity provider token endpoint. Verify the token endpoint URL is publicly reachable over https and that it accepts the configured client credentials. | | PRQL-BBGU | Import Artifact Staging Error | Prequel could not stage the batch file for delivery. The records were not sent and will be retried on the next load. | | PRQL-NSCH | Import Dataset Schema Unavailable | The dataset schema is unavailable until an extract has successfully extracted rows for this dataset. | | PRQL-MLAI | Import Delivery Outcome Integrity Error | Prequel detected an internal inconsistency while recording delivery results. The affected records were not marked delivered and will be retried on the next load. Please contact support if this persists. | | PRQL-QPMF | Import Destination Bad Gateway | The destination endpoint returned HTTP 502. Undelivered records are retried on the next scheduled load. | | PRQL-JPDQ | Import Destination Bad Request | The destination endpoint rejected the record with HTTP 400. Check the record data and destination configuration. | | PRQL-IMEK | Import Destination Client Error | The destination endpoint rejected the record with a client error (4xx). Check the destination configuration and record data. | | PRQL-KTQN | Import Destination Connection Refused | The destination endpoint refused the connection. Verify the URI and that the endpoint is accepting connections. | | PRQL-YCHU | Import Destination Context Cancelled | The delivery operation was cancelled before all records could be delivered. Undelivered records will be retried on the next load cycle. | | PRQL-FVRM | Import Destination DNS Error | DNS resolution failed for the destination endpoint. Verify the hostname in the destination URI. | | PRQL-CNJK | Import Destination Forbidden | The destination endpoint returned HTTP 403. Verify webhook signature configuration and credentials. | | PRQL-PRHQ | Import Destination Gateway Timeout | The destination endpoint returned HTTP 504. If the response carried a Retry-After header the request was retried immediately, otherwise the record will be retried on the next scheduled load. | | PRQL-QFJW | Import Destination Headers Too Large | The destination endpoint returned HTTP 431. Reduce the number or size of custom headers in the request template. | | PRQL-PHOK | Import Destination Internal Server Error | The destination endpoint returned HTTP 500. Undelivered records are retried on the next scheduled load. If the error persists, check the destination endpoint health. | | PRQL-TWER | Import Destination Invalid Acknowledgement | The destination acknowledgement payload was malformed or failed schema validation. | | PRQL-HWLT | Import Destination Network Error | A network error occurred while delivering a record to the destination endpoint. Check the destination URI and network connectivity. | | PRQL-VFTB | Import Destination Not Found | The destination endpoint returned HTTP 404. Verify the URI in the request template. | | PRQL-QAHC | Import Destination Payload Too Large | The destination endpoint returned HTTP 413. Reduce the record size or adjust the destination's payload limits. | | PRQL-WQKF | Import Destination Rate Limited | The destination endpoint rate-limited the request (HTTP 429). If the response carried a Retry-After header the request was retried immediately, otherwise the record will be retried on the next scheduled load. | | PRQL-EZDQ | Import Destination Request Error | Failed to construct the HTTP request for the destination endpoint. Check the destination configuration. | | PRQL-TNXG | Import Destination Server Error | The destination endpoint returned a server error (5xx). Undelivered records are retried on the next scheduled load. If the error persists, check the destination endpoint health. | | PRQL-HNNX | Import Destination Service Unavailable | The destination endpoint returned HTTP 503. If the response carried a Retry-After header the request was retried immediately, otherwise the record will be retried on the next scheduled load. | | PRQL-XNWB | Import Destination TLS Error | TLS handshake failed with the destination endpoint. Verify the endpoint supports HTTPS and has a valid certificate. | | PRQL-BXPM | Import Destination Template Error | The destination request template is misconfigured. Verify that the body template syntax is valid and all referenced fields exist in the destination's record schema. | | PRQL-GMVJ | Import Destination Timeout | The request to the destination endpoint timed out. Undelivered records are retried on the next scheduled load. If the error persists, check the destination endpoint latency. | | PRQL-NROT | Import Destination Unprocessable Entity | The destination endpoint returned HTTP 422. The record data is syntactically valid but semantically incorrect. | | PRQL-NBSI | Import Job Cancelled | The import job was cancelled. Please retry — or contact support if you did not initiate this and the issue persists. | | PRQL-VXPK | Import Primary Key Not String | The primary key column must be of type string (text/varchar). Please ensure the source table primary key column is string-typed. | | PRQL-KVZK | Import Provider Not Configured | No enabled import provider is available for this request. Create one with `POST /import/providers` or enable an existing provider before creating datasets, extracts, or loads. | | PRQL-WCKH | Import Transform Invalid Configuration | A stream mapping configuration was invalid or missing required fields like 'name' or 'target\_field'. | | PRQL-BKDX | Import Transform Invalid Payload | The transformation pipeline encountered an invalid JSON payload. Please ensure that stream records are valid JSON. | | PRQL-TRUN | Import Transform Runtime Error | The transformation pipeline encountered one or more runtime errors. Please review the mapping expressions and input data. | | PRQL-NOTD | Import Transform Unsupported Mapping | An unsupported mapping type was used in stream configuration. Supported types are 'field' and 'expr'. | | PRQL-VKHA | Import Transform Validation Failed | A stream record failed validation against the destination record schema after transformations were applied. | | PRQL-OHHG | Import Worker Unavailable | No import worker is currently available to process this request. The system will auto-recover shortly. Please retry. | | PRQL-INUM | Invalid Input: Numerical Type | One or more values could not be cast to a numerical type. Please ensure all source data columns of numerical types contain valid numerical values. Contact support if this error persists. | | PRQL-FGAD | Invalid Timestamp Value in Source Column | An invalid timestamp value was detected in a source column. Please review your timestamp columns for invalid values. | | PRQL-XTQZ | Missing Datalake Cache Resource | The dataset's cached data was not found in its datalake. Ensure the dataset belongs to a provider whose datalake bucket remains active and intact, and try again. | | PRQL-CLNU | MongoDB Authentication Failed | The MongoDB connection was refused due to insufficient permissions. Please check that the provided connection principal has been granted sufficient access. | | PRQL-KWZN | MongoDB Authentication Failed | The MongoDB connection was successful, but the user is not authorized. Please check that the provided user has access. | | PRQL-MJUM | MongoDB Authentication Failed | The MongoDB connection was refused. Please check that the provided MongoDB access details are correct. | | PRQL-FBNN | MongoDB Collection Not Found | The provided MongoDB collection was not found. | | PRQL-BGZS | MongoDB Connection Failed | The MongoDB connection was unsuccessful due to an unexpected configuration error. If the error persists, please contact support. | | PRQL-KDCW | MongoDB Connection Refused | The MongoDB connection was refused. Please check that the provided MongoDB server is valid and accessible. | | PRQL-MDXE | MongoDB Runtime Encountered Error | The MongoDB connection exceeded a space quota. | | PRQL-VIPD | MongoDB Runtime Encountered Error | The MongoDB connection encountered a memory limit. | | PRQL-OPXQ | MotherDuck Duplicate Primary Key Error | The transfer could not be completed because the source data contains more than one row with the same primary key value for this model. Please check the source table duplicate primary key values, deduplicate if needed, and try again. | | PRQL-QYYA | MySQL Authentication Failed | The MySQL connection was refused due to insufficient permissions. Please check that the provided connection details have been granted sufficient access. | | PRQL-AZBE | MySQL Connection Failed | The provided MySQL host was not found. Please check the the provided host name is valid and accessible. | | PRQL-DQTJ | MySQL Connection Failed | The provided MySQL port is invalid. | | PRQL-EWAS | MySQL Insufficient Create Table Permissions | The writer user does not have sufficient permissions to create tables in the destination schema. Please grant CREATE on the destination schema to the configured user and try again. | | PRQL-VMTE | MySQL Insufficient User Permissions | One or more permissions on the database are missing from the writer user. Please ensure all required permissions are provided and retry. | | PRQL-WCLX | MySQL Invalid Syntax Error | A SQL syntax error was encountered. Please contact support for more information. | | PRQL-AREW | Object Storage Connection Failed | The specified bucket name contains invalid characters. | | PRQL-EGJN | Object Storage Connection Failed | The specified bucket does not exist. | | PRQL-SAGK | Object Storage List IO Timeout | Object Storage client encountered IO timeout listing bucket objects. If this error persists, please contact support. | | PRQL-NQFC | Oracle Authentication Failed | The provided Oracle credentials are invalid. Please check that the correct username and password have been provided and try again. | | PRQL-WPMU | Oracle Connection Failed | The provided Oracle port is invalid. Please check that the provided port is a valid TCP port (typically 1521). | | PRQL-VRDH | Oracle Database Not Found | The Oracle listener at the provided host does not recognize the configured database. Please check that the provided database name is correct and try again. | | PRQL-OXKB | Oracle Host Not Found | The provided Oracle host was not found. Please check that the provided host name is valid and accessible. | | PRQL-CIUS | Oversized Decimal Value | Encountered a value of type DECIMAL that is too large to be converted to DECIMAL(38,19). | | PRQL-MCOQ | Postgres Authentication Failed | The Postgres connection was refused. Please check that the provided connection details are correct. | | PRQL-JXOS | Postgres Conflict With Recovery | The Postgres connection was dropped by the server. This is usually due to a Hot Standby related query timeout. If this error persists, please contact support. | | PRQL-RXOI | Postgres Connection Failed | The Postgres connection was refused. Please check that the provided host is valid and accessible. | | PRQL-TPOQ | Postgres Connection Failed | The Postgres connection was refused. Please check that the provided host is valid and accessible. | | PRQL-VLMV | Postgres Connection Failed | The Postgres connection was refused. Please check that the provided host is valid and accessible. | | PRQL-QJXV | Postgres Data Processing Failed | The Postgres connection encountered an error. At least one expected column was not found. | | PRQL-SMHC | Postgres Data Processing Failed | The Postgres connection encountered an error. At least one expected relation was not found. | | PRQL-XLSX | Postgres Data Processing Failed | The Postgres data transfer encountered duplicate primary key values. Please check that the primary key column values are unique for this recipient. | | PRQL-FLQR | Postgres Missing PostGIS Extension | The PostGIS extenstion is required for the Postgres to recognize the type `geometry`. Please make sure PostGIS is installed. | | PRQL-KKIM | Postgres Runtime Encountered Error | The Postgres connection encountered an unexpected error and was dropped. If this error persists, please contact support. | | PRQL-KPFZ | Postgres Runtime Encountered Error | The Postgres connection was dropped by the server. If this error persists, please contact support. | | PRQL-OKZP | Postgres Runtime Encountered Error | The Postgres connection was refused on the schema. Please check that the provided connection principal has been granted sufficient access. | | PRQL-UKMC | Postgres Runtime Encountered Error | The Postgres connection was dropped due to a conflict with the recovery instance. This is usually due to a Hot Standby related query timeout. If this error persists, please contact support. | | PRQL-ZSRS | Postgres Runtime Encountered Error | The Postgres connection was refused on the view. Please check that the provided connection principal has been granted sufficient access. | | PRQL-PGST | Postgres Statement Timeout Exceeded | The query exceeded the statement\_timeout configured on the Postgres database. Please raise statement\_timeout for the connecting role/user: [https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STATEMENT-TIMEOUT](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STATEMENT-TIMEOUT). | | PRQL-ROTX | Postgres cannot execute write in read-only transaction | Postgres is unable to execute the write query. Please ensure that Postgres is configured upon a writer endpoint and with write permissions. | | PRQL-QDFS | Postgres table missing replica identity | Postgres is unable to update a destination table because it is part of a logical replication publication but has no replica identity configured. Please ensure that any published tables have a primary key, or are set to REPLICA IDENTITY FULL, or exclude them from the publication. | | PRQL-RSLM | Record Size Limit Exceeded | One or more records in the source data exceed the maximum allowed size for import. Please reduce the size of these records or contact support for assistance. | | PRQL-DXHS | Redshift Authentication Failed | The provided Redshift connection was refused due to insufficient permissions. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-RSFA | Redshift Authentication Failed | The Redshift connection was refused. Please check that the provided connection principal has been granted sufficient access. | | PRQL-BOWD | Redshift Authorization Failed | The provided Redshift query failed due to insufficient database permissions. Please check that the provided connection details are correct and the principal has been granted sufficient permissions to the target database. | | PRQL-XKWZ | Redshift Cannot Drop Table with Dependencies | A DROP table operation failed due to one or more downstream consumers dependent on it. Please temporarily revoke access from downstream consumers to the destination table for the full refresh transfer to succeed. | | PRQL-NFCL | Redshift Cluster Not Found | The specified Redshift cluster could not be found in the target region. Please check that the cluster identifier is correct and the staging bucket is located in the same AWS region as the Redshift cluster. | | PRQL-WKQZ | Redshift Concurrent Write Conflict | A Redshift write aborted due to a conflict with a concurrent transaction on the same table. Please ensure only a single data export writer targets this schema and no other workloads are attempting to update the table. | | PRQL-BLHI | Redshift Connection Failed | The Redshift connection was refused. Please check that the provided host is valid and accessible. | | PRQL-FTNX | Redshift Connection Failed | The Redshift connection was refused. Please check that the provided host is valid and accessible. | | PRQL-ZJSM | Redshift Connection Reset By Peer | The Redshift connection was reset. This is likely due to a network issue. Please check your SSH, firewall, and network settings. If the issue persists, please contact support. | | PRQL-CSNQ | Redshift Data Processing Failed | The Redshift connection encountered an unknown error, contact support. | | PRQL-JCVI | Redshift Data Processing Failed | The Redshift connection encountered an error. At least one expected column was not found. | | PRQL-KMNN | Redshift Data Processing Failed | The Redshift connection encountered an error. At least one expected relation was not found. | | PRQL-XVKM | Redshift IAM Role Token Audience Mismatch | AWS rejected the federated identity token because its audience did not match what the IAM role expects. Please verify the oaud condition in the role's trust policy matches the provided value. | | PRQL-WUZB | Redshift Runtime Encountered Error | The Redshift connection encountered an unexpected error and was dropped. If this error persists, please contact support. | | PRQL-RSHA | Redshift Unable to Assume IAM Role | The Redshift connection was denied permission to assume the configured IAM role. Please verify the role's trust policy allows sts:AssumeRoleWithWebIdentity from the service identity and try again. | | PRQL-MWKP | S3 IAM Role Token Audience Mismatch | AWS rejected the federated identity token because its audience did not match what the IAM role expects. Please verify the oaud condition in the role's trust policy matches the provided value. | | PRQL-HMCK | S3 Invalid AWS Role ARN | The provided AWS IAM Role ARN is invalid. Please verify the ARN format and value is correct and try again. | | PRQL-BKDC | S3 Invalid Hostname | The connection contains an invalid hostname. Please verify your bucket region is a valid AWS region code (e.g. eu-west-1). | | PRQL-JSZM | S3 Missing Delete Permission | Unable to upload file to S3. Please check that the provided role has been granted s3:DeleteObject. | | PRQL-GWUV | S3 Missing List Bucket Permission | Unable to list objects in the S3 destination bucket. Please grant the s3:ListBucket permission to the configured role on the bucket. | | PRQL-RTTA | S3 Missing Upload Permission | Unable to upload file to S3. Please check that the provided role has been granted s3:PutObject. | | PRQL-CIAC | S3 Tables Missing Get Table Metadata Location Permission | A query failed due to a missing s3tables:GetTableMetadataLocation permission in the user/role policy. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-QLDD | S3 Tables Missing Get Table Permission | A query failed due to a missing s3tables:GetTableBucket permission in the user/role policy. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-GXDD | S3 Tables Missing Update Table Metadata Location Permission | A query failed due to a missing s3tables:UpdateTableMetadataLocation permission in the user/role policy. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-QGQQ | S3 Unable to Assume IAM Role | The S3 connection was denied permission to assume the configured IAM role. Please verify the custom trust policy on the role allows sts:AssumeRoleWithWebIdentity from the service identity, and that the required oaud/sub conditions are set correctly, and try again. | | PRQL-AWJZ | SFTP Connection Lost | SFTP client experienced a connection loss. Please review any idle timeout or session limits configured on the SFTP server and retry. | | PRQL-ANLW | SQL Server Connection Could Not Be Established | The SQL Server connection timed out. Please verify the host is reachable, firewall rules and/or security group allows access by whitelisting the static egress IP, and the port is reachable. | | PRQL-GZDZ | SQL Server Connection Failed | The provided SQL Server host was not found. Please check that the provided host name is valid and accessible. | | PRQL-VKDT | SQL Server Data Processing Failed | The SQL Server data transfer encountered duplicate primary key values. Please check that the primary key column values are unique for this recipient. | | PRQL-UXPW | SQL Server Invalid Username/Password Characters | The transfer user's username or password contains special characters that cannot be parsed during connection. Please update the user's credentials and try again. | | PRQL-LCNW | SQL Server Size Quota Exceeded | The database has reached its size quota. Please check the database service tier storage limit and any maximum dataset size configured, and increase allocations as needed. | | PRQL-MURC | SQL Server unable to resolve collation conflict | A collation conflict cannot be resolved on the SQL Server table. Please contact support for more information. | | PRQL-CLDD | Schema Not Found Error | A query failed due to a missing schema. Please check that the specified schema exists in the catalog or metastore. | | PRQL-ZPLL | Service Temporarily Unavailable | Prequel's import service is temporarily unavailable. Please retry shortly. If the issue persists, contact support. | | PRQL-FKDS | Snowflake Authentication Error | Snowflake user access is disabled. Please contact the Snowflake admin to get more information. | | PRQL-VKCP | Snowflake Authentication Failed | The Snowflake engine refused the operation due to insufficient permissions. Please check that the provided principal has been granted the required permissions. | | PRQL-ZBSP | Snowflake Authentication Failed | The provided Snowflake credentials are invalid. Please check that the correct credentials have been provided. | | PRQL-BJKE | Snowflake Authentication Policy Rejected | Snowflake rejected the login because the current authentication policy does not allow the attempted authentication method. Ensure the user/account policy allows DRIVERS and the chosen authentication method (PASSWORD or KEYPAIR). | | PRQL-IGRW | Snowflake Authentication Rate Limited | The Snowflake connection was rejected. This is likely due to too many failed connection attempts. Please check that the correct credentials have been provided. | | PRQL-EXLI | Snowflake Connection Failed | IP whitelisting has not been properly configured in the provided Snowflake account. Please contact support to confirm the correct IP address to whitelist within Snowflake. | | PRQL-HFCO | Snowflake Connection Failed | The provided Snowflake Account Host is not valid. Please check that your provided host name is a valid Snowflake account URL (ending in 'snowflakecomputing.com'). | | PRQL-PKAN | Snowflake Data Processing Failed | The Snowflake connection has no active warehouse. Please ensure the provided user has a default warehouse configured. | | PRQL-AFWL | Snowflake Database Does Not Exist or Not Authorized | The connection attempt failed due to an issue with permissions. Please confirm that the provided Snowflake database exists and the provided user has been authorized. If the error persists, please confirm that the provided user has been successfully granted access to the designated role, and that the default\_role has been successfully configured on the user. | | PRQL-NSOE | Snowflake Duplicate Row Error | The Snowflake data transfer encountered an error during data transfer. The Snowflake target table or staged data may have duplicate data preventing successful updates. Please confirm there is no duplicate data in the data source, and contact support if the issue persists. | | PRQL-BIEM | Snowflake Host Does Not Exist | The Snowflake host was unreachable. Please ensure the host was properly entered. | | PRQL-WLAM | Snowflake Incorrect Username or Password | The provided Snowflake username or passsword is incorrect. Please check that the correct username and password have been provided. | | PRQL-MVKR | Snowflake Integer Overflow | A value in a Snowflake integer column exceeds the maximum supported 64-bit integer size. | | PRQL-JWCP | Snowflake Invalid Account Setting | Failed to read the information schema due to casing. If the Snowflake account has the setting QUOTED\_IDENTIFIERS\_IGNORE\_CASE enabled, make sure it is set to FALSE on the data transfer user. | | PRQL-COSD | Snowflake Invalid Identifier | A Snowflake query tried to reference a table or column that does not exist. If the table or column has been deleted, it may need to be recreated manually or via full refresh. Please contact support for more information. | | PRQL-OFWE | Snowflake Key Auth Error | The Snowflake connection was refused. Please confirm that the provided Snowflake public key has been successfully assigned to the Snowflake user. | | PRQL-AJHJ | Snowflake Multi-factor Authentication is Required | Multi-factor authentication is currently required for the authorized user and MFA is not supported with data syncing. Please ensure your user is of type SERVICE instead of HUMAN and try again. | | PRQL-VIED | Snowflake No Such Host Error | The Snowflake connection failed to initialize, the host provided does not exist. | | PRQL-YWUX | Snowflake Object Verification Error | The Snowflake server was unable to verify an object. Please check that the Snowflake principal has access to the required objects. | | PRQL-UPSD | Snowflake Password Expired | The given Snowflake password has expired. Snowflake suggests updating the password via the Snowflake web console. | | PRQL-QZQZ | Snowflake Quota Exceeded | The designated Snowflake warehouse cannot be resumed because the warehouse's resource monitor has exceeded its credit quota. | | PRQL-IRPW | Snowflake Runtime Encountered Error | The Snowflake account has run out of virtual warehouse trial credit. | | PRQL-JWOD | Snowflake Runtime Encountered Error | Snowflake reported that the statement was no longer executing (error 000605). This usually means the query was canceled before it could run, which is typically transient and resolves automatically. If the error persists, check whether the target warehouse is overloaded and scale it up if queries are queuing. | | PRQL-WTKQ | Snowflake Runtime Encountered Error | The Snowflake account has been suspended due to the lack of a payment method. | | PRQL-SSTE | Snowflake Session Token Expired | The Snowflake session token has expired. Please ensure the configured warehouse is active and not overloaded, and retry the transfer. | | PRQL-SNOA | Snowflake Stage is Not Authorized | The connection encountered an issue accessing a Snowflake stage. If you have existing FUTURE STAGE grants at the database or schema level, please ensure that the transfer user's default role is provided GRANT READ and GRANT WRITE ON FUTURE STAGES in the target schema. | | PRQL-DSIO | Snowflake Timeout | Snowflake client encountered statement or warehouse timeout and canceled query. | | PRQL-BKEK | Snowflake Unknown Internal Error | Snowflake returned an internal error. Review the failed query in Snowflake's query history and check the Snowflake status page to determine whether the issue is ongoing. | | PRQL-DICX | Snowflake Warehouse Encountered Permissions Error | The Snowflake warehouse encountered an error. Please check that the provided service account has been granted sufficient permissions. Specifically, please check for permissions on existing tables and schemas. | | PRQL-KFSS | Source Circuit Breaker Tripped | The job encountered too many source mapping, transform, or validation errors and triggered the circuit breaker. Execution was halted early to protect system resources. Please review the failed records to determine the root cause. | | PRQL-DCBT | Stream Circuit Breaker Tripped | Deliveries for this stream failed persistently and its circuit breaker was tripped. Loads for this stream will be skipped until the circuit breaker is reset, which you can do by updating the stream once the underlying failure is addressed. | | PRQL-ANCT | Transfer Dispatch Error | The transfer failed to be dispatched to a worker. | | PRQL-IDWD | Trino Exceeded Partitions Limit | The Trino engine encountered an error because too many partitions were found. | | PRQL-ERMS | Unknown Error | An unknown error occurred. | | PRQL-XFVC | Unsupported Table Type | The specified table type is not supported for this datalake operation. | # Coded error handling Source: https://docs.prequel.co/export/error-handling/error-handling Understand the error objects returned by Prequel transfers and connection tests. ## Transfers and connection tests **Sensitive Traces** In accordance with data policy and security guarantees, Prequel also offers [Sensitive Trace Webhooks](/export/error-handling/sensitive-trace-webhook) to provide unmodified errors directly to your infrastructure. Follow the link to learn how to receive full errors from your Prequel environment, or continue reading about in-application coded errors below. When a transfer or connection test encounters an issue, an error object is returned. This object includes an error code, a message, and other fields to help identify the problem and provide steps to resolve it. **Understanding Our Terminology** * **First Party**: You, the Prequel user. * **Third Party**: Your customers. ### Error format in transfers and connection tests When an error occurs during a transfer or connection test, the response will include an `error` object with the following format: ```json title="Error object" icon="brackets-curly" expandable theme={null} { "error": { "error_code": "PRQL-PVER", "title": "BigQuery Authentication Failed", "blame": "third_party", "documentation_url": "https://docs.prequel.co/export/error-handling/error-codes", "message": "The BigQuery connection was refused. Please check that the provided connection principal has been granted sufficient access.", "trace": "...unable to ping bigquery database via service account impersonation..." // Full trace of the error } } ``` #### Blame The `blame` indicates who is responsible for the error and can have one of the following values: * `first_party`: The error is caused by the Prequel user (you). * `third_party`: The error is caused by your customer. * `prequel`: The error is caused by Prequel. * `host`: The error is caused by whoever is hosting Prequel (our cloud or your deployment). * `unknown`: No specific blame assigned. **Unknown Blame** We cannot anticipate (nor have we encountered) every error that the third party systems may raise. Error codes and messages are frequently changing and errors that we've encountered before should appear with a label and message. However, errors we have not encountered before will appear without a blame, and Prequel support can help you diagnose them. #### Error code, title, and message The `error_code`, `title`, and `message` fields provide more specific information about the error. * `error_code`: A short code that uniquely identifies the error. * `title`: A user-facing title for the error. * `message`: A user-facing message that explains the error. Refer to the [Error Codes](/export/error-handling/error-codes) page for a full list of error codes. ### Trace The `trace` field contains the full trace of the error. This can be useful for debugging the error. We don't recommend surfacing this to users. ### Connection test response format **Understanding `status`** There is a top level `status` field that indicates the status of the **request itself**. The inner `status` field, within the `data` object, indicates the status of the connection test. This allows you to distinguish between an error that occurs when communicating with the Prequel API and an error that occurs during the connection test. The `data` object on the connection test response contains the results of the connection test, assuming the request itself was successful. It will have a status of `success` or `error`. #### Example: connection test failed due to third-party user error ```json title="Connection test failure" icon="brackets-curly" expandable theme={null} { "status": "success", "data": { "status": "error", "success": false, "error": { "error_code": "PRQL-PVER", "title": "BigQuery Authentication Failed", "blame": "third_party", "documentation_url": "https://docs.prequel.co/export/error-handling/error-codes", "message": "The BigQuery connection was refused. Please check that the provided connection principal has been granted sufficient access.", "trace": "...unable to ping bigquery database via service account impersonation...", // Full trace of the error } } } ``` #### Example: connection test succeeded ```json title="Connection test success" icon="brackets-curly" expandable theme={null} { "status": "success", "data": { "status": "success", "success": true, } } ``` ### Transfer response format Errors on transfers can be found in `model_metrics`. This is a list of objects corresponding to the models being transferred. Prequel reports errors per-model. #### Example: partial transfer failure ```json title="Partial transfer failure" icon="brackets-curly" expandable theme={null} { "status": "success", "data": { "transfer": { ... "model_metrics": [ { ... "error": { "error_code": "PRQL-PVER", "title": "BigQuery Authentication Failed", "blame": "third_party", "documentation_url": "https://docs.prequel.co/export/error-handling/error-codes", "message": "The BigQuery connection was refused. Please check that the provided connection principal has been granted sufficient access.", "trace": "...unable to ping bigquery database via service account impersonation..." // Full trace of the error } } ] } } } ``` # Sensitive trace webhooks Source: https://docs.prequel.co/export/error-handling/sensitive-trace-webhook Configure webhooks to receive sensitive traces to your infrastructure. ## Overview As a matter of company policy and compliance, Prequel never stores or retains any of the data it transfers. Prequel occasionally encounters errors during data transfers, typically due to issues with the source or destination database systems. When these rare errors occur, the trace surfaced by the relevant database driver may occasionally include data included in the transfer. Note that this is entirely outside of Prequel's control. To uphold its data privacy and security guarantees, Prequel handles errors as follows: 1. In the Prequel application and Prequel API endpoints, Prequel surfaces a [coded error message](/export/error-handling/error-handling) which will **never** contain sensitive customer data. Raw traces are explicitly redacted from these messages to guarantee that no sensitive data is surfaced. 2. Prequel provides **Sensitive Trace Webhooks** to enable customers to receive **unmodified errors** in their own infrastructure standardized for a generic HTTP POST endpoint, providing additional context for debugging errors while keeping sensitive data out of standard logs. These payloads may include sensitive information. Verify your receiver is authenticated, access-controlled, and monitored according to your policies. ## When are sensitive traces sent? * Error details are sent **per-transfer**, aggregating errors across models in an array of traces. These are delivered as a single POST request per endpoint configured. * If no sensitive traces occur during a transfer, **no POST request is sent**. * Sensitive traces are also sent for **source and destination connection tests**. Sensitive traces are not forwarded for model validation errors. * One POST request is made per-transfer **to each sensitive trace endpoint** configured. *** ## Configuration steps Before creating the endpoint in Prequel, verify your receiver is ready to accept POST requests. This can be any HTTPS endpoint you control: an internal service, a Slack Incoming Webhook, a Datadog HTTP endpoint, or similar. If you are routing sensitive traces to Slack, you can use the generic Prequel webhook guide to configure your receiver: * [Slack Integration](/export/monitoring/slack) Use the [Create Sensitive Trace Endpoint](/export/api-reference/sensitive-trace-endpoints/create-sensitive-trace-endpoint) API to register your receiver with Prequel. ```text title="Create endpoint request" icon="file-lines" theme={null} POST /sensitive-trace-endpoints ``` Select an example request body to expand it: ```json title="Generic webhook body" icon="brackets-curly" expandable theme={null} { "sensitive_trace_endpoint": { "name": "Production Error Webhook", "description": "Sends errors to our internal monitoring system", "url": "https://your-domain.com/webhooks/prequel-errors", "secret": "your-webhook-secret-token", "header_template": "Authorization: Bearer {{.Secret}}\nContent-Type: application/json", "body_template": "{\"environment\": \"{{.Environment}}\", \"timestamp\": \"{{.Timestamp}}\", \"destination_url\": \"https://YOUR_ADMIN_HOST/destinations/{{.DestinationId}}\", \"errors\": {{.SensitiveTraces}}}" } } ``` ````json title="Slack webhook body" icon="brackets-curly" expandable theme={null} { "sensitive_trace_endpoint": { "name": "Slack Sensitive Traces", "description": "Sends sensitive traces to Slack via Incoming Webhook", "url": "https://hooks.slack.com/services/YOUR_WEBHOOK_URL", "header_template": "Content-Type: application/json", "body_template": "{\"attachments\": [{\"text\": {{printf \"*Prequel Sensitive Trace | Destination:* %s\\n\\n**\\n\\n*Environment:* %s\\n\\n*Timestamp:* %s\\n\\n*Traces:*\\n```json\\n%s\\n```\" .DestinationId .DestinationId .Environment .Timestamp .SensitiveTraces | printf \"%q\"}}, \"color\": \"danger\", \"mrkdwn_in\": [\"text\"]}]}" } } ```` Replace `YOUR_WEBHOOK_URL` with your Slack Incoming Webhook URL and `YOUR_ADMIN_HOST` with your Prequel admin console host (e.g. `app.prequel.co`). Notes: * The `header_template` is **plain text** with one header per line as `Key: Value` as shown in the example, not JSON. * For authorization, deliveries are not signed by Prequel, unlike other Prequel [webhooks](/export/monitoring/webhooks). You must authenticate using your secret, as shown. * To update or delete the endpoint at any time, use the [PATCH](/export/api-reference/sensitive-trace-endpoints/update-sensitive-trace-endpoint) and [DELETE](/export/api-reference/sensitive-trace-endpoints/delete-sensitive-trace-endpoint) endpoints respectively. *** ## Template system The URL, headers, and body of the webhook can be customized using Go `text/template` syntax. The following variables are available: | Variable | Type | Description | | ---------------------- | --------------- | --------------------------------- | | `{{.Secret}}` | `string` | Endpoint's stored secret for auth | | `{{.Environment}}` | `string` | `prod` or `staging` | | `{{.Timestamp}}` | `string` | RFC3339 timestamp | | `{{.SensitiveTraces}}` | `string` (JSON) | JSON array of trace objects | | `{{.DestinationId}}` | `string` | Destination ID for the transfer | | `{{.SourceIds}}` | `string` (JSON) | JSON array of source IDs | | `{{.ModelIds}}` | `string` (JSON) | JSON array of model IDs | ### Default behavior When no custom `header_template` or `body_template` is provided, Prequel uses the following defaults: **Default Headers** ```text title="Default headers" icon="file-lines" theme={null} Content-Type: application/json Authorization: Bearer {Secret} ``` **Default Body** ```json title="Default body" icon="brackets-curly" expandable theme={null} { "environment": "prod", "sensitive_traces": [ { "destination_id": "dest-123", "trace": "ERROR: ..." } ], "destination_id": "dest-123", "timestamp": "2024-01-15T10:30:00Z" } ``` *** ## Sensitive trace payload schema Each item in `SensitiveTraces` is an object that can include: ```json title="Trace object schema" icon="brackets-curly" expandable theme={null} { "destination_id": "dest-123", "source_id": "src-456", "model_id": "model-789", "model_name": "users_table", "source_vendor": "postgres", "destination_vendor": "snowflake", "execution_context": "write_destination", "trace": "ERROR: duplicate key value violates unique constraint ..." } ``` For connection tests, `execution_context` will reflect the test (e.g., `test_source_connection` or `test_destination_connection`). `trace` is always present. *** ## Related Subscribe to event types and configure delivery methods. Route Prequel events into a Slack channel. Forward Prequel events into Datadog. Review the events Prequel records through the logs API. # Append-only mode Source: https://docs.prequel.co/export/features/append-only-mode Optimize incremental writes for append-only data models ## Overview Append-only is an optimization mode for data models where your records are insert-only with no updates after initial write. With this guarantee, Prequel narrows the scope of work for each transfer and optimizes table layout in supported destinations, improving performance during transfer. Use append-only mode when your upstream system only appends new events/rows. Append-only is configured on one or more [data models](/export/concepts/models) you maintain in Prequel. **Support** Append-only mode is supported for the following destinations: BigQuery, Snowflake, Redshift, Databricks, Delta Lake, and Athena. Object storage (S3, GCS, ABS) and SFTP behave as insert-only by default, where files are written and no merge operations are performed. ## How it works * Prequel uses your model's primary key and last-modified column to incrementally upsert new data. * With append-only mode enabled, Prequel makes an assumption that no existing rows with timestamps prior to the latest `updated_at` value have been updated. This reduces the search space (by partitions) from which Prequel must locate and update rows in the destination. This improves write speed to the destination. * In supported destinations, Prequel also finetunes clustering/partitioning more narrowly to align storage layout with recent data and query patterns. For more detail on clustering and partitioning support by destination type, **contact the Prequel team** ## Configuration Enable append-only mode per model by setting `is_append_only` to `true` on model with a PATCH model request ([API reference](/export/api-reference/models/update-model)). **Switching to append-only mode requires a full refresh** Switching any model to append-only mode requires a [full refresh transfer](/export/concepts/transfers) for the change to take effect, which reloads all historical data for that model. Contact the Prequel team if you are interested in migrating to append-only mode. ## Append-only behavior & interactions * Transfers remain upserts; new rows will be inserted and existing rows (e.g., prior rows included in a transfer's lookback window) are merged based on primary key. * Transfers that move records already present in the destination (e.g., a transfer re-triggered with a custom window, or repeated rows resulting from the lookback window), **do not create duplicates in warehouse or database destinations**. Rows will be MERGED into the destination as long as the row's `updated_at` timestamp **has not been modified**. * Append-only works with source queries, multi-tenant filtering, and windowed transfers. ## Considerations * Ensure that your records are not modified, including any enrichment jobs **after** landing in the source that Prequel points to. * Use your Prequel staging environment to test, validate performance, then migrate in production. ## See also Configure the columns and config files for each model. Browse supported destination types. Limit how much history each transfer moves. Roll out column and type changes to destinations safely. # Change detection Source: https://docs.prequel.co/export/features/change-detection Understanding Prequel's change detection logic ## Change detection logic At a high level, Prequel relies on a `last_modified_at` type column to detect changes between transfers. On the first transfer to any destination (or on a `full_refresh`), all historical data is transferred, and the greatest `last_modified_at` value is recorded. On any subsequent transfer, the data is filtered such that only data with greater than any of the previous `last_modified_at` values is transferred. This allows Prequel to predictably transfer batches of updated data. ```sql title="Incremental transfer query" icon="database" expandable theme={null} -- simplified incremental transfer query SELECT * FROM some_source_table WHERE some_last_modified_at_column >= to_timestamp({{.LastModifiedAtEpoch}}) AND some_organization_id_column = '{{.IdInProviderSystem}}'; ``` **Tip: Use arrival time, not event time, when possible as your `last_modified_at` column** Prequel's high watermark is computed by calculating the greatest `last_modified_at` value seen in the last transfer. Rows with a `last_modified_at` at or below the watermark are not re-queried on the next incremental transfer. **Arrival time** (when the row was written to your source database, e.g., `updated_at` set to `NOW()` by your application on every insert or update) is generally preferred as the `last_modified_at` column choice. Regardless of when the underlying event occurred, the row's `last_modified_at` reflects when it landed in your source, so it will always be above the current watermark and included in the next transfer. This prevents possible out of order data arrival issues. ## Timestamp precision When the greatest `last_modified_at` value is recorded, it is stored as an epoch, or "unix timestamp". This value has second precision. For example, if a batch of data is transferred where the greatest `last_modified_at` timestamp value was `2025-01-01 1:15:30 AM`, the equivalent epoch integer will be stored: `1735694130`. ## Eventual consistency In many cases, the source data platform may have eventual consistency concerns. This is especially true in cases where the `last_modified_at` timestamp is generated by an external system and cannot be guaranteed to be inserted as monotonically increasing values. For this reason, Prequel adds a "lookback window" to each incremental transfer. This "lookback" window differs by source vendor: | Source Vendor | Lookback Window | | ------------------------------ | ------------------- | | Athena | 5 minutes | | PostgreSQL / Aurora PostgreSQL | 30 seconds | | MySQL / Aurora MySQL | 5 seconds | | All other vendors | 5 minutes (default) | ```sql title="Lookback window query" icon="database" expandable theme={null} SELECT * FROM some_source_table -- lookback window varies by vendor (e.g., 5 min for Athena) WHERE some_last_modified_at_column >= to_timestamp({{.LastModifiedAtEpoch}} - INTERVAL 5 MIN) AND some_organization_id_column = '{{.IdInProviderSystem}}'; ``` ## Duplication behavior in Prequel How Prequel handles duplicate data depends on your destination type: ### Database and data warehouse destinations **No duplicates in your tables.** Prequel uses upsert operations to ensure that lookback windows do not create duplicate rows in database destinations like Snowflake, BigQuery, PostgreSQL, etc. ### Object storage destinations **Duplicates may occur.** Object storage destinations (S3, GCS, Azure Blob Storage) are append-only by design, unless using [Delta Lake](/export/destinations/delta-lake). Lookback windows may cause the same data to be written multiple times and it is the expectation that these are deduplicated by the downstream data pipeline after data is written to the destination. ### Primary key requirements For all destinations, your source data must have unique primary key values within each transfer batch. Duplicate primary keys in your source data will cause transfer errors. ## Interaction with a custom `source_query` If you choose to use a `source_query` instead of the default table query, you may wonder how your source query interacts with the change detection queries. In cases where a `source_query` is used, Prequel applies the same predicate filtering outside of the custom `source_query`. ```sql title="Filtering over a source query" icon="database" expandable theme={null} WITH some_source_query AS (...) SELECT * FROM some_source_query WHERE some_last_modified_at_column >= to_timestamp({{.LastModifiedAtEpoch}} - INTERVAL 5 MIN) AND some_organization_id_column = '{{.IdInProviderSystem}}'; ``` **Change detection over a `source_query`** If your `source_query` preemptively filters data using a `last_modified_at` column, you may avoid the safety of the lookback window. In some cases, this may be intentional, but this tradeoff should be considered as you think about your upstream data pipeline. **Timestamp precision in a `source_query` resulting in missing rows** Because Prequel only stores the `LastModifiedAtEpoch` with **second** precision, comparing that value with a timestamp of higher precision could result in skipped rows. For example, imagine a scenario where: * Prequel observes an `last_modified_at` time of `2025-01-01 00:00:01.600`. * Prequel records this as an Epoch `1735689601` (corresponding to `2025-01-01 00:00:02`). * A custom `source_query` is used, which does not account for rounding/second precision: * `SELECT * FROM some_source_query WHERE some_last_modified_at_column >= to_timestamp({{.LastModifiedAtEpoch}}` Since the custom `source_query` does not **also** round the `some_last_modified_at_column` to second level precision, any rows of data with values timestamps between `00:00:01.600` and `00:00:02.00` may be skipped. This can easily be solved by either rounding the `some_last_modified_at column`, or implementing your own lookback window. ## Rows transferred and delta rows In the Prequel UI (Transfer details) and API responses, you'll often see two row counts: * **Rows** (`rows_transferred`): How many rows were selected from the source for this model in that transfer. * **Delta rows** (`delta_rows_transferred`): An estimate of how many rows in that transfer have a `last_modified_at` value **newer than the most recent previously completed transfer** for that model (the current high watermark). ### `rows_transferred`: how it's calculated `rows_transferred` is an **extraction metric**: it counts the rows returned by Prequel's source query. * The lower bound is **inclusive**. * The upper bound is **exclusive** when an end window is specified. * A **lookback window** is applied by querying slightly earlier than the lower bound; rows from this overlap are included in `rows_transferred`. ```sql title="Rows transferred count" icon="database" expandable theme={null} -- simplified rows_transferred query (count of extracted rows) SELECT COUNT(*) FROM some_source_table WHERE some_last_modified_at_column >= to_timestamp({{.LastModifiedAtEpoch}} - INTERVAL 5 MIN) AND some_last_modified_at_column < to_timestamp({{.LastModifiedAtEpochUpperBound}}) AND some_organization_id_column = '{{.IdInProviderSystem}}'; ``` **Notes on bounds** The exact lookback duration depends on the source vendor (see the table above). If `end_transfer_window_at` is not set, the upper-bound predicate may be omitted. ### `delta_rows_transferred`: how it's calculated `delta_rows_transferred` is a **calculated estimate** derived from the extracted rows. It counts rows whose `last_modified_at` is **strictly newer** than the current high watermark. * Included: rows where `last_modified_at > high watermark` (and within the transfer's upper bound, if one is specified). * Excluded: the lookback overlap (to avoid counting that overlap in delta rows). ```sql title="Delta rows count" icon="database" expandable theme={null} SELECT COUNT(*) FROM internal_data_batch WHERE CAST({{.LastModifiedAtColumnName}} AS TIMESTAMP WITH TIME ZONE) > CAST(to_timestamp({{.HighwaterMarkEpoch}}) AS TIMESTAMP WITH TIME ZONE); ``` * **Why it can be `0` for historical transfers**: If you run a transfer over an older time range, `delta_rows_transferred` can be `0` even when `rows_transferred` is large if none of the rows in that window have a `last_modified_at` value newer than the most recent completed transfer's high watermark. # Change tracking Source: https://docs.prequel.co/export/features/change-tracking Tracking transfer completion at the destination After each successful transfer, Prequel writes a change tracking artifact to the destination. This allows downstream pipelines and your customers to determine when data was last synced for each model. The artifact type depends on the destination: | Destination type | Change tracking artifact | | :-------------------------- | :-------------------------------------------------------------------------------------- | | Database / data warehouse | [`_transfer_status` table](/export/features/transfer-status-table) | | Object storage | [Manifest files for object storage](/export/features/manifest-files-for-object-storage) | | Spreadsheet (Google Sheets) | Not available | # Connection modalities Source: https://docs.prequel.co/export/features/connection-modalities Understanding Prequel's various source and destination connection options | Connection option | Description | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Username & password** | By default, most sources and destinations are accessed via single-purpose users. These users can be used in combination with other connection options (like SSH tunneling, IP whitelisting, or RBAC accounts) for enhanced security. | | **Service accounts** | Where available, Prequel also supports accessing sources and destinations via special purpose service accounts. | | **SSH tunneling** | Some sources and destinations are not exposed to the public internet, but are instead accessed via an SSH tunnel through a bastion host in the same VPC. Prequel supports SSH tunneling on sources and destinations that support it. | | **IP whitelisting** | Some sources and destinations support enhanced access control by restricting access to an IP whitelist. In most deployments, the Prequel service will be deployed behind a set of static IPs that can be whitelisted for this purpose. | | **Role-based access control (RBAC) accounts** | In some cases, the owner of a multi-tenanted source or destination may wish for a given tenant's data to be accessed with a tenant-specific user. For these cases, Prequel supports using tenant-specific users and credentials to access data.

**Note:** using this method will incur additional operational overhead as a new user will need to be generated for each additional connection. | | **Key-Pair** | Prequel can generate public/private key pairs, and supported source & destination systems can whitelist the public key for passwordless auth. | # Source queries Source: https://docs.prequel.co/export/features/custom-queries Using your own source queries Prequel has the optional ability to write your own [`source_query`](/export/api-reference/models/update-model) with access to some of the important Prequel variables that will be templated in at runtime. For example, in the Prequel configuration, you could write: ```json title="Source query example: deduplication" icon="brackets-curly" expandable theme={null} { ..., "source_query": "SELECT * FROM ( SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY updated_at DESC) AS rank_order FROM . WHERE tenant_id = {{.IdInProviderSystem}} AND updated_at >= from_unixtime({{.LastModifiedAtEpoch}}) ) WHERE rank_order = 1", ... } ``` Prequel will still run a query on top of the `source_query` (to handle column selecting and renaming), so the `columns` section of the config will not change. This change simply allows you to be more expressive around how Prequel reads data, and provides the option to specifically leverage some of Prequel's parameters (like the epoch timestamp) in your query, enabling efficient partition pruning, etc. Source query configuration is parsed as JSON5, so comments (`//` and `/* */`) and line continuation characters (`\`) are supported. Use these to document complex queries inline. `source_table` is required in the configuration when using the custom query feature. `source_table` is a schema-qualified reference to the source table (e.g., `source_schema.table_name`). In schema-tenanted mode, the schema qualifier is used only for model validation; transfers use the recipient's `id_in_provider_system` as the schema. See [Tenancy options](/export/concepts/multi-tenancy). ## Available Prequel variables | Available Variable | Data Type | Description | | :----------------------------------- | :-------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `{{.LastModifiedAtEpoch}}` | `integer` | The greatest (most recent) epoch Prequel has transferred so far. Sample value: `1681841760` | | `{{.LastModifiedAtEpochUpperBound}}` | `integer` | The upper bound epoch (exclusive) for the current transfer. Set when the transfer specifies an end window (for example, a transfer request with `end_transfer_window_at`, or each window of a [windowed transfer](/export/features/windowed-transfers)); | | `{{.IdInProviderSystem}}` | `string` | Tenant ID (available in "multi-tenant table" mode) | | `{{.Schema}}` | `string` | Schema name (available in "schema-tenanted database" mode). Resolves to the recipient's `id_in_provider_system`. | | `{{.PrimaryKeyColumnName}}` | `string` | The name of the primary key column in the source table. Sample value: `id` | | `{{.LastModifiedColumnName}}` | `string` | The name of the last-modified timestamp column in the source table. Sample value: `updated_at` | | `{{.OrganizationColumnName}}` | `string` | The name of the tenant/organization column in the source table (available in "multi-tenant table" mode). Sample value: `tenant_id` | ## Use cases ### Skipping the lookback window for batched timestamps By default, Prequel queries with a [lookback window](/export/features/change-detection#eventual-consistency) before the last transferred epoch to catch late-arriving rows. When the `last_modified` column is batched (many rows share the same timestamp), this lookback can unintentionally pull a large set of already-transferred rows on each transfer. To eliminate the lookback, filter directly on `{{.LastModifiedAtEpoch}}` in your source query: ```json title="Source query example: skip lookback window" icon="brackets-curly" expandable theme={null} { ..., "source_query": "SELECT * FROM .
WHERE tenant_id = {{.IdInProviderSystem}} AND {{.LastModifiedColumnName}} >= from_unixtime({{.LastModifiedAtEpoch}})", ... } ``` ### Using source queries with parameterized views Source queries can also be used to support parameterized views in sources that support the functionality (e.g., ClickHouse). ```json title="Source query example: parameterized view" icon="brackets-curly" expandable theme={null} { ..., "source_query": "SELECT * FROM some_parameterized_view(tenant_id = {{.IdInProviderSystem}})", ... } ``` ### Distributing a shared dataset to multiple recipients If your source table has no tenant column, you can distribute the same data to multiple recipients by combining the `organization_column` field with a source query. Set `organization_column` to a column name that does not exist in the source table (e.g., `org_id`), then define a source query that injects each recipient's ID as that column: ```json title="Source query example: inject recipient ID column" icon="brackets-curly" expandable theme={null} { ..., "organization_column": "org_id", "source_query": "SELECT *, {{.IdInProviderSystem}} AS org_id FROM .
", ... } ``` At runtime, Prequel substitutes each recipient's `id_in_provider_system` for `{{.IdInProviderSystem}}`, creating a virtual `org_id` column used to route data. `org_id` does not need to be listed under `columns` - recipients will not see it in their tables. ### Pre-aggregating metrics If your source table contains raw event rows, you can pre-aggregate them in the source query so recipients receive summary data directly, without needing to aggregate on their end. ```json title="Source query example: pre-aggregated metrics" icon="brackets-curly" expandable theme={null} { ..., "source_query": "SELECT {{.IdInProviderSystem}} AS tenant_id, DATE_TRUNC('day', event_time) AS event_date, event_type, COUNT(*) AS event_count, SUM(value) AS total_value FROM events WHERE tenant_id = {{.IdInProviderSystem}} GROUP BY DATE_TRUNC('day', event_time), event_type", ... } ``` In this case, none of the output columns exist in the source table. All columns returned by the source query must be listed in the model config's columns, otherwise they will not be selected during transfer. ## Limitations * DDL statements (`CREATE`, `DROP`, `ALTER`, etc.) are not supported. * The query must return a valid result set that Prequel can query on top of and that matches all `columns` specified in the model. # Data integrity checks Source: https://docs.prequel.co/export/features/data-integrity-checks Auditing the data sent to your destinations One of Prequel's core promises is the guarantee that data in the source will be accurately replicated in the destination. Every row will be transferred, no row will be dropped, and the destination will get all changes to existing rows. We want every user to build that confidence for themselves instead of having to take our word for it. This is why we surface data integrity checks, a way to audit the integrity of the data present in the destination, and compare it to what's in the source. ## Results Data integrity checks return a range of information about the underlying destination table. Some results are informational, while others denote data integrity issues. | Field | Action needed if greater than 0 | Meaning | | :--------------------- | :-------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sampling_rate` | None, this is expected. | Bounded from 0 to 1. A value of 1.0 indicates that no sampling was used. This number represents the percentage of (primary key, `last_modified_at`) tuples compared within partitions with differing hashes. | | `total_rows_compared` | None, this is expected. | Indicates the number of (primary key, `last_modified_at`) tuples in the source that were compared with the destination. | | `total_rows_pending` | None, this is expected. | The number of rows present in the source that will be inserted into the destination with the next transfer. | | `total_rows_stale` | None, this is expected. | The number of rows present in the source that have changed since they were last modified in the destination, and that will be updated in the destination with the next transfer | | `total_rows_missing` | Full refresh the destination and contact Prequel support. | The number of rows present in the source that should exist in the destination but do not. | | `total_rows_incorrect` | Full refresh the destination and contact Prequel support. | The number of rows present in the source that exist in the destination but do not have the correct `last_modified_at` value. | | `total_rows_orphaned` | None. | The number of rows that are present in the destination but not in the source. This is usually the result of hard deletes in the source, or of foreign data having been introduced in the destination by the recipient. | ## Limitations Data integrity checks work by comparing hashes of certain values between the source and the destination. This allows them to be relatively compute and bandwidth efficient. In order to work across systems and in a way that doesn't put undue load on those systems at scale, they are subject to a handful of limitations. ### Supported systems Data integrity checks are currently only supported on the following systems: * Athena * BigQuery * Databricks * Postgres * Redshift * Snowflake ### Other limitations * Data integrity checks only ensure the integrity of primary keys and `last_modified_at` timestamps. This is sufficient to build assurance of integrity. * On large tables, integrity checks leverage random sampling to build probabilistic confidence in the integrity of the data. This prevents the checks from overloading either system, and allows them to run in a finite time. * Automated sampling will likely occur on tables receiving more than 1B rows per day in volume. * You can also leverage manual sampling and specify a set of primary keys to compare across systems (regardless of table size). * Data Integrity jobs will not run at the same time as transfers to the same destination and vice versa. This is to prevent overloading the resources of the source or destination. This is also to ensure that integrity measurements are not taken on a "moving target". If a data integrity check is in progress, all transfers to that destination will remain in PENDING state until the integrity check is complete. # Manifest files for object storage Source: https://docs.prequel.co/export/features/manifest-files-for-object-storage Per-model transfer manifests written to object storage destinations Object storage destinations (`s3`, `gcs`, `abs`, and `s3_compat`) receive manifest files alongside the data files. This enables any downstream pipeline processing this data to know when a transfer has completed for a given model. A manifest file is written **for each model** as part of every transfer. Manifest files live in their own `_manifests` directory within the object storage destination, and each manifest file is named `manifest_{transfer_id}.json` as shown below. These files are enabled by default on new object storage destinations. ## File format Every manifest file follows the same structure. Here is a list of keys you can expect in every file, along with a sample complete file. | Object Key | Value | | :--------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `version` | Manifest format version. | | `transfer_id` | Unique ID of the transfer job. | | `start_time` | Start time of the transfer job in UTC time. | | `end_time` | End time of the transfer job in UTC time. | | `model_id` | Unique ID (uuid) of the data model. | | `model_name` | Name of the data model. | | `bucket_name` | Name of the object storage bucket. | | `bucket_prefix` | Prefix of all objects created in this transfer. | | `manifest_file_key` | Key of the manifest file (path and file name). | | `file_format` | Format of the landed data (eg `PARQUET`). | | `transfer_type` | Type of data transfer. Either `FULL_REFRESH` or `INCREMENTAL`. | | `signature` | SHA-256 signature of the value stored in the `files` key. | | `signature_public_key` | Public key which can be used to read the signature. | | `files` | Array of file objects. Each object contains: a `sha256_checksum`, the `etag` of the file, and the `data_file_key` (the full path and name the file was written to). | ```json title="Sample manifest file" icon="brackets-curly" expandable theme={null} { "version": "2024-06-01", "transfer_id": "aeb6efc1-73ec-405d-ae5e-d28b349b364c", "start_time": "2024-06-01T07:28:34.028Z", "end_time": "2024-06-01T07:33:43.897Z", "model_id": "ee9542e3-6469-4e09-bdcc-abb50ca5643a", "model_name": "transactions", "bucket_name": "vendor-data", "bucket_prefix": "user-supplied-schema/transactions/2024-06-01", "manifest_file_key": "user-supplied-schema/transactions/2024-06-01/manifest_aeb6efc1-73ec-405d-ae5e-d28b349b364c.json", "file_format": "PARQUET", "transfer_type": "FULL_REFRESH", "signature": "f4d2e40ab3c0f5b8e3b88e022db4f7c54fb9f82c77ffa2e444c479b57843f57bea32812a73b9c8a786d3b908c434f9374e0498b9a2f23c90e2f578b9444382b0f", "signature_public_key": "-----BEGIN RSA PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDmf2CGFxZU/Dx911t4K8l/G5zM\njGUvhP01k2YTLtBRXEdXLGZnmzuJTOsqyPOvj3+HU/iNUQ/mXIJu7wKTrA/glZ1i\n0Zcc18Ek0jyne03ikBDIdyeYZTGi37/UnNVLwkr2FxhHUHBgiS5msFjxjquC941D\n5Xluak1U1p6/ZFV0AwIDAQAB\n-----END RSA PUBLIC KEY-----", "files": [ { "sha256_checksum": "sQMSpEILNgoQmarvDFonGQ==", "etag": "af83d6f217c19b8b0fff8023d8ca4716-1", "data_file_key": "user-supplied-schema/transactions/2024-06-01/20240601150301.parquet" }, { "sha256_checksum": "9c78d2e727b9f0b56a85b38dff88763c==", "etag": "9f84f7aacc09e05-1", "data_file_key": "user-supplied-schema/transactions/2024-06-01/20240601150405.parquet" }, { "sha256_checksum": "1a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p==", "etag": "3d4fgsd7834f734b-1", "data_file_key": "user-supplied-schema/transactions/2024-06-01/20240601150645.parquet" }, ] } ``` # Orchestration Source: https://docs.prequel.co/export/features/orchestration How Prequel schedules transfers and how to tune frequency per destination. Prequel automatically orchestrates transfers on a schedule. Every account runs at a default frequency, and Prequel enqueues a transfer for each enabled destination at that cadence with no manual intervention. ## Default orchestration Every Prequel account runs at a default frequency, configurable at the account level. Unless otherwise specified, this default value is daily (i.e., every 24 hours). **Orchestration time of day is global** The time of day at which scheduled transfers run is global to your account. For example, daily transfers run at midnight UTC and 12h transfers run at midnight and noon UTC. Contact the Prequel team if you would like to update this global orchestration time. ## Per-destination frequency You can override the default frequency for an individual destination, either in the Prequel Admin UI or via the API by setting `frequency_minutes` on the [Create Destination](/export/api-reference/destinations/create-destination) (or [Update Destination](/export/api-reference/destinations/update-destination)) request. ![](https://storage.googleapis.com/prequel_docs/images/ui-sync-frequency.png "Sync frequency selection menu as seen in Prequel Connect and the Admin UI") ### Allowed frequencies The allowed frequencies are every **15m**, **1h**, **6h**, **12h**, or **daily**. Available sync frequencies depend on your plan. Contact your Prequel representative to enable higher frequencies. | Frequency | `frequency_minutes` | Notes | | -------------------- | ------------------- | -------------------------------------------------------------- | | Organization default | `0` | Uses the organization's default frequency | | 15m | `15` | | | 1h (hourly) | `60` | | | 6h | `360` | 6h transfers will run every 6 hours starting midnight UTC time | | 12h | `720` | 12h transfers will run midnight & noon UTC time | | 24h (daily) | `1440` | Daily transfers will run midnight UTC time | ## Custom orchestration For scheduling needs beyond the standard frequencies above, you can configure custom orchestration directly in the Prequel UI. If you would like to disable automatic orchestration entirely (for example, to trigger every transfer yourself through the API), contact the Prequel team for more detail. # Partner attribution Source: https://docs.prequel.co/export/features/partner-attribution Attribute Prequel workloads to your vendor partnerships **Availability** Partner attribution is available on self-hosted and private cloud hosted Prequel packages only. Contact the Prequel team for more information. ## Overview Prequel works closely with vendor partners to ensure you can easily satisfy your partner relationships, including partner attribution on the workloads Prequel runs. When enabled, Prequel identifies itself with a partner-specific identifier on every connection it opens to a supported vendor. The queries and API calls Prequel makes on your behalf are then attributed to your partnership, including transfers to destinations owned by your customers. ## Supported partners | Partner | How it is sent | Identifier | | ------------------------------------------- | -------------------------------------------- | ---------------------------------------- | | Google Cloud (BigQuery, GCS, Google Sheets) | `User-Agent` header on all API calls | `{partner}_Prequel/1.0 (GPN:{partner};)` | | Snowflake | `application` connection parameter | `{partner}_Prequel` | | Databricks | `User-Agent` entry on the SQL driver | `prequel+{partner}/2024.1` | | ClickHouse | Client product info on the native connection | `prequel+{partner}/3.0` | For self-hosted deployments, set the `collie.host_name_for_partner` value in your helm chart to your partner identifier. Prequel formats it according to each vendor's requirements. For private cloud deployments, contact the Prequel team to have your deployment configured and verified for partner attribution. If you have questions about partner attribution, or would like support for a partner program not listed here, contact the Prequel team. # Schema evolution Source: https://docs.prequel.co/export/features/schema-evolution How Prequel evolves destination schemas safely as your data model changes ## Overview As your product data and data model changes over time, so too may the shape of the data you wish to share with customers. As your schema changes, Prequel can help you evolve the data model schema in destination systems. When you update your data model in Prequel, schemas in destination systems will be updated automatically on the next transfer to that destination. Prequel's schema evolution capabilities are limited to a set of "safe" and non-destructive operations that should handle most schema evolution use cases without resulting in undesirable consequences in the destination system. **Model configuration validation enforces safe schema evolution** Only model updates that result in permitted schema evolution operations are permitted. Non-allowed model updates will be rejected. | Evolution | Behavior in Destination for Rows Arriving in the Future | Behavior in Destination for Previously Loaded Rows | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Add column** | Column is added to all existing destinations and created in new destinations. Column is populated on all subsequent transfers. | Rows inserted prior to the addition of this column will not retroactively populate this column. These rows will be `NULL` up until the transfer that performed the evolution.
*Note: if new columns on previously transferred rows need to be populated, a transfer can be submitted with a specific time window to backfill missing data.* | | **Delete column** | On existing destinations, rows inserted after the deletion of this column will no longer populate this column and will result in `NULL` values. On new destinations, column is no longer added. | Column will remain on previously inserted rows. (Column will not be retroactively deleted.) | | **Add table** | Model is transferred to all applicable destinations on the next scheduled transfer. Because it will be the first transfer for this model per destination, the table will be created and backfilled automatically. | *N/A* | | **Delete table** | Model is no longer transferred to any destinations. | On existing destinations, the table is no longer updated (*but will not be deleted*). | ## Limitations * **Changing data types** is not allowed. Many data destinations do not allow changes to data types. To avoid broken destination states, Prequel prevents changing data types on existing columns. * **Reusing table names within the same destination** is not allowed. Two models assigned to the same destination must write to different tables. Models assigned to different destinations may share a table name. * **Changing the `name_in_destination` of the primary key or last modified timestamp column** is not allowed. * **Column position is not guaranteed** When a column is added to an existing model, it is appended to the end of the destination table schema via `ALTER TABLE ADD COLUMN`, regardless of where it appears in the model config. Downstream pipelines should not be dependent upon column positioning. ## How it works Schema evolutions happen automatically. When a transfer is enqueued, the data model configured at that time (`submitted_at`) is recorded and assigned with that transfer. When the schema is dequeued (usually \<1 minute later), the necessary schema evolutions are calculated and performed (e.g., new columns are added) before the latest batch of data is loaded into the destination. The need for schema evolution is determined by comparing the schema from the previous successful evolution to the schema in enqueued transfer. Any permissible schema evolution operations will then be performed to modify the destination schema as needed. # Transfer status table Source: https://docs.prequel.co/export/features/transfer-status-table Per-model transfer completion timestamps written to database and data warehouse destinations Database and data warehouse destinations receive a `_transfer_status` artifact in the destination schema after each successful transfer. It contains one row per model with the timestamp of the last successful sync. ## Schema | Column | Type | Description | | :------------------------- | :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `data_model_name` | `string` | Name of the data model (matches the destination table name). | | `transfer_last_updated_at` | `timestamp` | Timestamp of the most recent completed transfer for this model. Note that new data may not have been written on the last successful sync (e.g., if there were no changes). | ## BigQuery For BigQuery destinations, `_transfer_status` is a **view** rather than a physical table. It is backed by BigQuery's `__TABLES__` metadata and a `_transfer_status_cache` table, which caches timestamps to prevent schema evolution operations from inflating the last-modified time. The `_transfer_status_cache` table is also present in the destination schema and is used internally by the view; it does not need to be queried directly. # Usage data Source: https://docs.prequel.co/export/features/usage-data Send your Prequel usage data to your own data warehouse ## Overview You can receive your Prequel usage data in your own data warehouse, using any of our supported [destination vendors](/export/destinations/overview). Contact Prequel support to enable and configure this feature. Once you configure your usage data destination, Prequel syncs three tables to your destination: `audit_log`, `destinations_snapshot`, and `model_transfers_log`. By default, this data is refreshed hourly. ## Data format Prequel syncs three tables to your destination: `audit_log`, `destinations_snapshot`, and `model_transfers_log`. ### `audit_log` Each row in this table corresponds to one event in the audit log, and is the same data available via the [`/logs` endpoint](/export/api-reference/monitoring-&-logging/list-logs). These logs are intended to be a comprehensive record of all write actions taken on your Prequel account. For more information, including the events included in this table, see the [audit log documentation](/export/monitoring/monitoring). | Field Name | Data Type | Description | | ------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `event_id` | `string` | Audit log ID. This ID is the same as the `/logs` endpoint. | | `occurred_at` | `timestamp` | Timestamp when event occurred, corresponds to `created_at` in logs endpoint, reported in UTC. | | `event_type` | `string` | The type of event. | | `resource_id` | `string` | The ID of the resource involved in the event. | | `actor_id` | `string` | The identifier of the actor who initialized the event. Some historical events have the value `unknown`. | | `actor_type` | `string` | The type of actor who initialized the event. The possible values are `unknown`, `user`, `api_key`, `recipient`. | | `current_resource` | `json` | Nullable JSON object representing the current state of the resource. DELETE operations will never have this field populated. Some historical events do not have this field populated. | | `previous_resource` | `json` | Nullable JSON object representing the previous state of the resource. CREATE operations will never have this field populated. Some historical events do not have this field populated. | ### `destinations_snapshot` Each entry in this table corresponds to one destination in your Prequel account. Note that this table represents the current state of destinations configured in Prequel. To get a historical record of actions taken on destinations, you can use the audit log data. | Field Name | Data Type | Description | | ------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------- | | `destination_id` | `string` | Destination ID in the Prequel API. | | `is_enabled` | `bool` | Whether scheduled syncs are enabled on this destination. | | `updated_at` | `timestamp` | Timestamp when the destination was last updated. | | `created_at` | `timestamp` | Timestamp corresponding to when the destination was first created. | | `recipient_id` | `string` | The ID of the recipient the destination is assigned to. | | `id_in_provider_system` | `string` | The ID in the source provider system, corresponds to the tenant schema in schema tenanting mode. | | `products` | `json` | JSON array of objects representing assigned products. Each entry has the fields `id` and `name`. | | `frequency_minutes` | `int` | The frequency at which Prequel's scheduler enqueues transfers to this destination. | | `enabled_models` | `json` | JSON array representing the enabled models for this destination. Each entry has the fields `id` and `name`. | | `has_all_models_enabled` | `bool` | Indicates whether the destination inherits all current and future models on its products. | | `destination_name` | `string` | The name of the destination. | | `destination_vendor` | `string` | The vendor of the destination. | | `max_concurrent_transfers` | `int` | Maximum number of simultaneous transfers allowed. | | `max_concurrent_queries_per_transfer` | `int` | Maximum number of concurrent queries allowed per transfer. | | `deleted_at` | `timestamp` | Timestamp when the destination was deleted. Null for active destinations. | ### `model_transfers_log` Each entry in this table corresponds to one completed model transfer within a transfer job. Model transfers that are pending or in progress are not currently included in this table. | Field Name | Data Type | Description | | -------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model_transfer_id` | `string` | Model ID concatenated with transfer ID. This ID only appears within Prequel's usage data. | | `transfer_id` | `string` | The ID of the transfer in the API. | | `model_id` | `string` | The ID of the model being transferred. | | `model_transfer_id` | `string` | A unique ID for the model transfer, combining transfer ID and model ID. | | `model_name` | `string` | The name of the model being transferred. | | `source_id` | `string` | The ID of the source from which the model was read. | | `source_name` | `string` | The name of the source from which the model was read. | | `source_vendor` | `string` | The vendor of the source from which the model was read. | | `destination_id` | `string` | The ID of the destination to which the model was written. | | `recipient_id` | `string` | The ID of the recipient for the destination. | | `id_in_provider_system` | `string` | The ID in the provider system, corresponds to the tenant schema. | | `destination_name` | `string` | The name of the destination. | | `destination_vendor` | `string` | The vendor type of the destination. | | `is_full_refresh` | `bool` | Indicates whether the transfer was a full refresh. | | `transfer_submitted_at` | `timestamp` | Timestamp when the transfer was submitted. | | `transfer_started_at` | `timestamp` | Timestamp when the worker began executing the model transfer. | | `transfer_ended_at` | `timestamp` | Timestamp when the model transfer completed. | | `transfer_start_window_at` | `timestamp` | Lower bound timestamp for the data transferred. | | `transfer_end_window_at` | `timestamp` | Upper bound timestamp for the data transferred. | | `status` | `string` | The status of the model transfer (e.g., `SUCCESS`, `ERROR`, `CANCELLED`). | | `enqueueing_actor_id` | `string` | The identifier of the actor who enqueued the transfer. | | `enqueueing_actor_type` | `string` | The type of actor who enqueued the transfer, taking one of the following values: `user`, `api_key`, `service`, or `unknown`. | | `error_trace` | `string` | Detailed error message in case of a transfer error. | | `error_code` | `string` | Error code corresponding to the transfer error. | | `error_blame` | `string` | The party responsible for the error. | | `error_message` | `string` | Error message for the model transfer. | | `documentation_url` | `string` | URL to documentation for further information on the error. | | `transfer_log` | `json` | Transfer log information. | | `billing_date` | `date` | The date on which Prequel will bill for this transfer. This is the date of the transfer `ended_at` at time zone America/Los\_Angeles. | | `rows_transferred` | `bigint` | Total number of rows selected from the source for this model during this transfer. | | `delta_rows_transferred` | `bigint` | Estimated number of rows in this transfer with a `last_modified_at` newer than the latest previously completed transfer for this model (does not include lookback overlap). | | `volume_transferred_in_mb` | `float` | The memory footprint of the data transferred. | **Warning** Deleting a model can cause historical transfer records to lose their associated model and source details. This issue is fixed for future transfers, but the time period of affected historical data varies by Prequel instance. Please contact support for more information. Transfers for deleted destinations may not be available in `model_transfers_log` prior to a certain date, depending on your deployment. For more details about transfers for deleted destinations, please contact support. ## Staging usage data Both production and staging environment data from Prequel is synced. Staging environment data is synced as separate tables to the same destination with a `_staging` suffix: * `audit_log_staging` * `destinations_snapshot_staging` * `model_transfers_log_staging` ## Example queries ```sql title="Rows transferred per day" icon="database" expandable theme={null} SELECT DATE(transfer_ended_at) AS transfer_date, SUM(rows_transferred) AS total_rows_transferred FROM `{{your-schema}}.model_transfers_log` WHERE destination_id = '{{your-destination-id}}' AND transfer_ended_at >= CAST(CURRENT_DATE - INTERVAL '1 MONTH' AS TIMESTAMP) GROUP BY transfer_date ORDER BY transfer_date; ``` ```sql title="Latest transfer status" icon="database" expandable theme={null} WITH LatestStatus AS ( SELECT model_id, MAX(transfer_ended_at) AS last_transfer_time, ARRAY_AGG(status ORDER BY transfer_ended_at DESC LIMIT 1)[OFFSET(0)] AS latest_status MAX(CASE WHEN is_full_refresh THEN transfer_ended_at ELSE NULL END) AS last_full_refresh_time FROM `{{your-schema}}.model_transfers_log` WHERE destination_id = '{{your-destination-id}}' GROUP BY model_id ) SELECT l.model_id, l.latest_status, l.last_transfer_time, l.last_full_refresh_time, CASE -- Have we ever triggered a transfer for this model? WHEN l.latest_status IS NULL THEN false ELSE true END AS transfer_has_been_triggered FROM LatestStatus l LEFT JOIN `{{your-schema}}.destinations_snapshot` d ON d.destination_id = '{{your-destination-id}}'; ``` ```sql title="Transfer metrics by billing date" icon="database" expandable theme={null} SELECT billing_date, destination_id, COUNT(*) AS count_transfers, COUNTIF(status = 'SUCCESS') AS count_transfers_succeeded, COUNTIF(status = 'ERROR') AS count_transfers_error, SUM(COALESCE(rows_transferred, 0)) AS total_rows_transferred, SUM(COALESCE(volume_transferred_in_mb, 0)) AS total_volume_transferred_mb, SUM(COALESCE(delta_rows_transferred, 0)) AS total_deltas_transferred, SUM(TIMESTAMP_DIFF(transfer_ended_at, transfer_started_at, SECOND)) AS total_transfer_time, SUM(TIMESTAMP_DIFF(transfer_started_at, transfer_submitted_at, SECOND)) AS total_queue_wait_time, SUM(TIMESTAMP_DIFF(transfer_ended_at, transfer_started_at, SECOND)) AS total_execution_time FROM `{{your-schema}}.model_transfers_log` GROUP BY billing_date, destination_id ORDER BY billing_date DESC; ``` ```sql title="Destination update history" icon="database" expandable theme={null} SELECT DATE(occurred_at) AS update_date, occurred_at AS update_timestamp, actor_id, actor_type, resource_id AS destination_id, previous_resource, current_resource FROM `{{your-schema}}.audit_log` WHERE event_type = 'DESTINATION_UPDATE' AND resource_id = '{{your-destination-id}}' ORDER BY occurred_at DESC; ``` # Windowed transfers Source: https://docs.prequel.co/export/features/windowed-transfers Automatically split up transfers into smaller chunks ## Overview By default, each transfer will transfer all available, newly updated data. There are cases where that behavior is not optimal: perhaps the volume of data is large, and the recipient does not want to wait for all the data to be backfilled before starting to consume it. It is possible to configure Prequel to split up any transfer into smaller windows of an arbitrary size. This configuration happens on a per-model basis. If a transfer fails during a given window, the next transfer will pick things up starting at that window again. In other words, progress from previous windows will be saved even in the case of an error. ## How it works To leverage windowed transfers, two new fields must be specified on each Prequel model: * `max_transfer_time_window_seconds`: this specifies the largest possible time increment Prequel will use when moving data for this model. For example, say this value is set to 1 day (`"max_transfer_time_window_seconds": 86400`), and a transfer is triggered that would transfer 10 days of data. This transfer will automatically be broken up into 10, one-day windows. * `min_start_time_epoch`: this specifies the time at which to start backfilling this model for any new destination. The easiest way to illustrate why this value is necessary is through an example. Let's assume that the `max_transfer_time_window_seconds` is set to 86400, aka one day. Each Prequel transfer will now try to move at most one day's worth of data. By default, Prequel will try to move data starting at epoch 0 (January 1 1970 at midnight UTC). Starting at that date, and moving data one day at a time, it will take over 19,000 windows to catch up to today. In other words, that might take a good long while and your source datastore might not be happy. The `min_start_time_epoch value` lets you specify a more reasonable starting point. What a good value is depends, among other things, on your own data. Perhaps your dataset starts January 1 2023. In this case, it will "only" take \~400 windows to catch up. Here is an example model definition which leverages those parameters. ```json title="Windowed model definition" icon="brackets-curly" expandable theme={null} { "model_name": "logs", "columns": ["..."], "max_transfer_time_window_seconds": 86400, "min_start_time_epoch": 1650000000, "source_table": "source_schema.application_logs", "source_name": "Example Production Source", "organization_column": "organization_id" } ``` **Recipient-level override** It is possible to override `min_start_time_epoch` at the recipient level. To do so, populate the `default_full_refresh_min_start_time_epoch` on the desired recipient via the [Update recipient](/export/api-reference/recipients/update-recipient) endpoint. If specified, this value will apply across all models (even ones that don't explicitly leverage windowed transfers). ## FAQ Queries for minimum values across large-scale datasets can be extremely expensive. # Building your customer experience Source: https://docs.prequel.co/export/getting-started/building-your-customer-experience The four ways to onboard your customers to Prequel Export, from self-serve magic links to a fully embedded SDK. Prequel supports four ways to onboard your customers' destinations. Use them on their own or together, depending on your product and team. Send a secure self-onboarding link. No code required. First-party guided onboarding, done by you in the Admin UI. Drop pre-built, themeable components into your React app. Build a fully custom experience against the Export API. ## Magic links Magic links offer a secure self-onboarding experience for your customers. When a customer opens their link, they get a pre-configured form scoped to their destination type and account, with a built-in connection test so they can validate their credentials before finishing. You provide a few details upfront to scope the link to a specific customer and destination. Magic links are single-use and expire after 7 days. See [Recipients](/export/concepts/recipients) for how they fit into onboarding. ## Prequel Admin UI Onboard a customer destination yourself, directly in the Prequel Admin UI, with no code. This suits first-party guided onboarding, ad-hoc setup, internal testing, or credential-less destinations where a self-serve flow is not needed. ![](https://storage.googleapis.com/prequel_docs/images/ui-add-snowflake-destination-form.png "\"Add Destination\" flow in the Admin UI") ## React SDK The React SDK provides pre-built, themeable components for destination onboarding and monitoring inside your React app. It handles tenant-scoped authentication, form rendering, and connection validation, so you can embed customer-facing data syncing without building it from scratch. See the [React SDK](/export/sdks/react/overview) and [Authentication](/export/sdks/react/authentication) guides. ![](https://storage.googleapis.com/prequel_docs/images/react-example-add-destination-form.png "Example of an \"Add Destination\" form powered by Prequel's React SDK.") ## API Build a fully custom experience against the [Export API](/export/api-reference/overview). The API covers the full lifecycle: creating and managing sources, destinations, recipients, and models, and triggering transfers from your own UI. ## Monitoring experience Give customers visibility into their syncs: which destinations exist, when the last transfer ran, and whether it succeeded. List the destinations a recipient has connected. Surface per-destination transfer history and status. For automated alerting on transfer outcomes, pair the in-product view with [webhooks and monitoring integrations](/export/monitoring/monitoring). # How Prequel Export works Source: https://docs.prequel.co/export/getting-started/how-prequel-export-works Connect a source, declare what is shared, and Prequel syncs it to each customer's destination on a schedule. Prequel Export moves data from a database, warehouse, or object store you control into your customers' own data stores, on a schedule, filtered to each customer's rows. You configure it once through the API or Admin UI, and Prequel handles extraction, type mapping, change detection, and delivery across vendors and connection types. ![](https://storage.googleapis.com/prequel_docs/images/prequel-illustrative-setup-steps.png) ## The building blocks The database, warehouse, or object store Prequel reads from. Declares which source table and columns are shared, how rows are filtered per tenant, and how changes are detected. One of your customers. Their rows are matched by a tenant identifier so they only receive their own data. Where a recipient receives their data. Owned by the recipient. [Models](/export/concepts/models) can be grouped into [products](/export/concepts/products) to control which subset of data each recipient receives. ## How a transfer works Prequel connects directly to your source and reads only the tables your models reference. Each model maps a source table to destination columns, with one column flagged as the primary key and one as the last-modified timestamp for change detection. Add a recipient for each customer and connect the destination they receive data into. The recipient's tenant identifier scopes every transfer to that customer's rows. The first transfer runs a full refresh; subsequent [transfers](/export/concepts/transfers) sync only changed rows on the configured [frequency](/export/features/orchestration), with [schema evolution](/export/features/schema-evolution) and [data integrity checks](/export/features/data-integrity-checks) handled automatically. Ready to build a pipeline end to end? Start with the [Quickstart guide](/export/getting-started/quickstart). # Quickstart guide Source: https://docs.prequel.co/export/getting-started/quickstart Configure a Prequel Export pipeline end to end and sync data to your first customer destination In this guide, you will configure a Prequel export from source to destination. Initial setup typically takes 1-2 hours to complete. ## Select your onboarding experience Get started with the Prequel agent skill, the Prequel MCP server, a Postman collection, or curl commands. Skills are an open standard for packaging reusable agent capabilities. Claude Code, Cursor, and other skill-aware agents can all load the Prequel Export skill. Open the [raw skill file](https://docs.prequel.co/.well-known/agent-skills/prequel-export-quickstart/SKILL.md), copy the contents, and save the file to your agent's skills directory. For Claude Code use `~/.claude/skills/prequel-export-quickstart/SKILL.md`. For Cursor use `.cursor/skills/prequel-export-quickstart/SKILL.md` in your project root and reload the workspace. Set up a new Prequel Export pipeline from scratch using the prequel-export-quickstart skill. Walk me through creating each resource in order, run the curl commands as we go, and finish by triggering a transfer to verify the pipeline end to end. The Prequel docs are available as an MCP server, exposing search tools your agent can call to look up the correct API operations and request schemas as it works. Point your AI tool at `https://docs.prequel.co/mcp`. The setup differs by client: ```bash title="Claude Code" icon="terminal" theme={null} claude mcp add --transport http prequel https://docs.prequel.co/mcp ``` ```json title="Cursor" icon="brackets-curly" expandable theme={null} // .cursor/mcp.json { "mcpServers": { "prequel": { "url": "https://docs.prequel.co/mcp" } } } ``` For full instructions, see the MCP setup guides for [Claude Code](https://docs.claude.com/en/docs/claude-code/mcp) and [Cursor](https://docs.cursor.com/context/model-context-protocol). Set up a new Prequel Export pipeline from scratch. Use the Prequel MCP server to look up the correct API operations and request schemas as we go, walk me through creating each resource in order, and finish by triggering a transfer to verify the pipeline end to end. The MCP server is read-only documentation search and does not act on your systems. If you pair it with the agent skill, which runs real API calls, review each command before running it and confirm before pointing an automated run at a customer's production environment. In Postman, click **Import**, paste the [Prequel Export OpenAPI spec URL](https://docs.prequel.co/generated/openapi-generated-2023-12-01.json), and confirm. Postman generates a complete collection from the spec with one request per operation. Add a `PREQUEL_API_KEY` variable to the imported collection and reference it from each request's `X-API-KEY` header. Browse the [API reference](/export/api-reference/overview) and select the operation for the resource you want to create. Use the **Copy** button on each operation's example request body, paste it into your shell as the `-d` payload, and substitute your values and resource IDs as you go. ## Prerequisites Access the Prequel Admin UI at [app.prequel.co](https://app.prequel.co), [eu-app.prequel.co](https://eu-app.prequel.co), or your custom domain. You can log in using **Continue with Google** if your company uses Google Workspace, or a one-time passcode sent to your email. If you are unable to log in, contact the Prequel team. You will need a Prequel API key. For a POC, you will use a staging key, which begins with `sk_staging`. The key in the `X-API-KEY` header routes your request to production or staging based on the key. Contact your Prequel representative if you do not have one. ```bash title="Set your API key" icon="terminal" theme={null} export PREQUEL_API_KEY="" ``` Every source table you want to transfer needs a unique ID column (e.g. `id`), a last-modified timestamp column (e.g. `updated_at`) for change detection, and a way to filter by tenant, either an organization ID column (e.g. `organization_id`) or a per-tenant schema. See [Change detection](/export/features/change-detection) and [Multi-tenancy](/export/concepts/multi-tenancy). If your source or destination restricts inbound traffic, allowlist the Prequel static IP before connecting. See [Prequel IP addresses](/export/deployment/prequel-ips). Confirm your key works against the API. A 200 response means your environment is ready. ```bash title="Verify API access" icon="terminal" theme={null} curl -H "X-API-KEY: $PREQUEL_API_KEY" https://api.prequel.co/sources ``` **Use the Staging environment** For initial onboarding, use the toggle in the bottom-left corner of the Admin UI to switch to the **Staging** environment. Your staging API key only creates and updates resources within the staging environment. When you are ready to begin work in your production environment, contact the Prequel team to receive a Production API key and flip to the **Production** environment in the Admin UI. ![](https://storage.googleapis.com/prequel_docs/images/ui-admin-environments-toggle.png) ## Setup overview A Prequel Export pipeline is built in five steps: 1. **Connect a source** so Prequel can read from your database, warehouse, or object store. 2. **Configure your data model** to declare which tables and columns are shared and how changes are detected. 3. **Add a recipient** to represent the customer you are sharing data with. 4. **Add a destination** that the recipient receives data into. 5. **Trigger a transfer** and verify data lands in the destination. ## Set up your pipeline Each call returns a resource ID in the response body. Record the IDs as you go, since later steps reference them with placeholders such as `` and ``. A source is the database, warehouse, or object store Prequel reads from. Create a new user or role in your source, granted read-only permissions on only the datasets you intend to export, to adhere to the principle of least privilege. Then use the per-vendor fields from the [source configuration guides](/export/sources/overview) to add it. See [Sources](/export/concepts/sources). Open **Sources → Add source**, choose your vendor, enter the connection details, and run the built-in connection test before saving. ![](https://storage.googleapis.com/prequel_docs/images/bigquery-add-source.png) ```bash title="Create source" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/sources \ -H "X-API-KEY: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": { "name": "postgres-read-replica", "vendor": "postgres", "host": "read-replica.db.example.com", "port": 5432, "database": "production", "user": "prequel", "password": "" } }' ``` ```bash title="Test connection" icon="terminal" theme={null} curl -X POST https://api.prequel.co/sources//test-connection \ -H "X-API-KEY: $PREQUEL_API_KEY" ``` A model declares which source table is shared, which columns map to the destination, how rows are filtered per tenant, and how changes are detected. Models are defined as config files: store them in a Git repository and publish them with the [GitHub Action](/export/integrations/github-action), or post them to the API directly. See [Models](/export/concepts/models). Posting a [model](/export/api-reference/models/create-model) validates it against the source unless `skip_model_validation` is set. **Is your source schema-tenanted?** Prequel environments are set to multi-tenant mode by default. If your source uses one schema per customer with identical tables across schemas (schema-tenanted), contact the Prequel team to switch your environment to schema-tenanted mode. See [Tenancy options](/export/concepts/multi-tenancy) to understand the difference between these models and confirm which is right for your environment. ```bash title="Create model" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/models \ -H "X-API-KEY: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": { "model_name": "events", "description": "Platform events shared with customers", "source_id": "", "source_table": "public.events", "organization_column": "organization_id", "columns": [ { "name_in_source": "id", "name_in_destination": "id", "data_type": "text", "is_primary_key": true }, { "name_in_source": "event_log", "name_in_destination": "event_log", "data_type": "text" }, { "name_in_source": "updated_at", "name_in_destination": "updated_at", "data_type": "timestamp", "is_last_modified": true } ] } }' ``` A recipient represents one of your customers. The `id_in_provider_system` is the tenant identifier Prequel matches against each model's `organization_column` to filter rows for that customer. In schema-tenanted mode, it is instead the tenant's exact schema name in the source. See [Recipients](/export/concepts/recipients). Open **Recipients → Add recipient**, give them a name and their tenant identifier (`id_in_provider_system`), and choose which [products](/export/concepts/products) they receive (or all of them). Pass `["all"]` for `products` to share every model, or a list of product names to share a subset. ```bash title="Create recipient" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/recipients \ -H "X-API-KEY: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "recipient": { "name": "Acme Corp", "id_in_provider_system": "acme", "products": ["all"] } }' ``` A destination is where a recipient receives their data. Use the per-vendor fields from the [destination configuration guides](/export/destinations/overview) and attach it to the recipient via the [Create destination](/export/api-reference/destinations/create-destination) endpoint. See [Destinations](/export/concepts/destinations). During initial configuration, we recommend first configuring an internal destination you can access, so you can review landed data as if you were one of your customers. To test a destination type you do not have access to, contact the Prequel team for a sandbox instance. Open **Destinations → Add destination**, select the recipient and vendor, and enter the connection details. You can open the destination's configuration documentation from the **Documentation** button in the top-right corner of the form. You can also collect destination credentials from your customer with a [magic link](/export/concepts/recipients). ![](https://storage.googleapis.com/prequel_docs/images/ui-add-snowflake-destination-form.png) ```bash title="Create destination" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/destinations \ -H "X-API-KEY: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "destination": { "name": "acme-snowflake", "recipient_id": "", "vendor": "snowflake", "host": "acme.us-east-1.snowflakecomputing.com", "database": "PREQUEL", "schema": "PUBLIC", "user": "TRANSFER_USER" } }' ``` With a source, model, recipient, and destination in place, [start a transfer](/export/api-reference/transfers/create-transfer). The first transfer runs a full refresh; subsequent transfers sync only changed rows. Open the destination and click **Sync now** to kick off a transfer. ![](https://storage.googleapis.com/prequel_docs/images/ui-sync-now-button.png) ```bash title="Trigger transfer" icon="terminal" theme={null} curl -X POST https://api.prequel.co/destinations//transfer \ -H "X-API-KEY: $PREQUEL_API_KEY" ``` That's it. Query the destination to confirm data arrived. The [`_transfer_status`](/export/features/transfer-status-table) table records the outcome of each model's transfer. Once connected, data automatically replicates to the destination on an ongoing basis, starting at the next scheduled frequency interval. To onboard each additional customer, repeat [Add a recipient](#add-a-recipient), [Add a destination](#add-a-destination), and [Trigger a transfer](#trigger-a-transfer). ## Explore integrating with your product When you are ready to move beyond manual setup, Prequel can be embedded into your product for a self-serve, monitored integration: Embed destination setup and transfer monitoring directly in your product UI. Publish and validate model changes from CI instead of by hand. Get notified on transfer success and failure via webhooks, Datadog, or Slack. Understand error codes and surface actionable failures to your customers. ## Next steps Roll out column and type changes to destinations safely. Tune how often each destination transfers. Verify source and destination row counts match. Choose how tenant data is modeled and filtered per recipient. # Overview Source: https://docs.prequel.co/export/index Prequel is the easiest way for B2B software companies to transfer data to customers. This guide is written exclusively for **Prequel Export**. To learn more about **Prequel Import**, see the [Prequel Import Guide](/import). Prequel makes it easy for engineering and product teams to connect to customers' data warehouses, databases, or object storage and sync data to them. Prequel connects directly to the data stores and efficiently moves data from source to destination. Complexity around high volumes, high reliability, and data integrity, across SQL dialects, data types, and connection modalities, is all abstracted away behind the Prequel API. Set up Prequel and share data with your first customer destination. Every Export API endpoint, with request and response schemas. ## Core concepts Where Prequel reads your data from. Define which data is shared and how it is shaped. Where your customers receive their data. The customers you share data with. How Prequel moves data on a schedule. Isolate and scope data per recipient. ## Supported destinations Prequel writes to all major warehouses, databases, object stores, and lakes. Load fully-typed tables into cloud data warehouses like Snowflake, BigQuery, Redshift, Databricks, and Athena. Write to the transactional databases your customers already run, including Postgres, MySQL, MongoDB, Oracle, and SQL Server. Deliver `Parquet`, `CSV`, or `JSON` in Apache Hive-style partitions to S3, Google Cloud Storage, Azure Blob Storage, and SFTP. Get native schema evolution, deduplication, and ACID transactions on your lake with Delta Lake and Apache Iceberg. **Support** Questions getting started? Contact your Prequel representative. For FAQ and how-to guides on using Prequel Export, see the [Prequel Export Knowledge Base](https://support.prequel.co). # GitHub Actions Source: https://docs.prequel.co/export/integrations/github-action Automate updates to your models and products from JSON config files in your GitHub repo. ## Overview Prequel offers GitHub Actions to automate updates to your **models** and **products** based on `.json` config files stored in your GitHub repo. Use these to: * Keep your model/product configs version-controlled * Automatically sync changes to Prequel on every commit * Define your data contract directly in code ## Supported actions You can use the following GitHub Actions provided by Prequel: * `prequel-co/apply-model-configs@v1`, upserts models * `prequel-co/apply-product-configs@v1`, upserts products These GitHub actions utilize the Prequel API and apply config files from your repo directly into your Prequel instance. **Note that the sample is set up to update configs for your `production` environment in Prequel.** To set up an action that will update `staging` configs, update the `branches` field to point to your staging branch, and pass it your `PREQUEL_STAGING_API_KEY` instead of the production one. ```yaml title="prequel_cd.yaml" theme={null} # This is a basic workflow to help you get started with Prequel Actions name: Prequel Continuous Delivery # Controls when the workflow will run on: # Triggers the workflow on push or pull request events but only for the "main" branch push: branches: [ "main" ] # Allows you to run this workflow manually from the Actions tab workflow_dispatch: jobs: sync: name: Sync with Prequel runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Upsert Models uses: prequel-co/apply-model-configs@v1 with: host: https://api.prequel.co api_key: ${{ secrets.PREQUEL_API_KEY }} mode: export dir: prequel/models/*.json - name: Upsert Products uses: prequel-co/apply-product-configs@v1 with: host: https://api.prequel.co api_key: ${{ secrets.PREQUEL_API_KEY }} mode: export dir: prequel/products/*.json ``` ## Model config format Model config files live in your repo. The default locations `prequel/models/*.json` and `prequel/products/*.json` can be overwritten with the `dir` field). Each file describes **one model** and is passed into the `apply-model-configs` action. Data model configuration is documented in full below. * [Data model configuration](/export/concepts/models#data-model-configuration) * [Model API Reference](/export/api-reference/models/create-model) ```json title="example_model_config.json" theme={null} { "model_name": "logs", "columns": [ { "name_in_source": "id", "name_in_destination": "id", "data_type": "text", "is_primary_key": true, "is_last_modified": false }, { "name_in_source": "log", "name_in_destination": "event_log", "description":"A descriptive text entry of the event that occurred.", "data_type": "text", "is_primary_key": false, "is_last_modified": false }, { "name_in_source": "updated_at", "name_in_destination": "updated_at", "data_type": "timestamp", "is_primary_key": false, "is_last_modified": true } ], "source_table": "source_schema.application_logs", "source_name": "Example Production Source", "organization_column": "organization_id" } ``` Tip: Using `source_name` instead of `source_id` provides stability across both staging and prod environments, whereas `source_id` is environment-specific. ## Product config format Product config files live in your repo (e.g. `prequel/products/*.json`) and define which models are grouped together into a "product" for sync. Each file defines **one product**. Product configuration is documented in more detail below. * [Product configuration](/export/concepts/products#getting-started-with-products) * [Product API Reference](/export/api-reference/products/create-product) ```json title="transactions_product.json" theme={null} { "models": ["transactions"] } ``` Currently, you cannot attach a model to a product **from the model config itself**, products must reference the model explicitly using this method. ## FAQ Yes. A common pattern is to override `model_name` based on branch name to prevent collisions. This can be done using custom scripting in your workflow or naming conventions in your files. These should live in your GitHub repo (e.g., `prequel/models/*.json`, `prequel/products/*.json`). They are pulled by the GitHub Action and applied to Prequel automatically. ## Additional resources * [Model API reference](/export/api-reference/models/create-model) * [Product API reference](/export/api-reference/products/create-product) * [Model overview](/export/concepts/models) * [Destination setup guides](/export/destinations/overview) # Datadog Source: https://docs.prequel.co/export/monitoring/datadog Use webhooks to monitor Prequel events directly in your Datadog. Use this guide to understand Prequel webhook events sent as Datadog Events. ## Overview Prequel supports sending webhook events directly to Datadog as events. When you create a webhook endpoint of type `datadog`, Prequel will POST a JSON payload to the [Datadog Events v1 API](https://docs.datadoghq.com/api/latest/events/#post-an-event), using your Datadog API key. The webhook can be easily configured directly in the Prequel Admin UI. ## Payload structure The payload sent to Datadog depends on the event type: * **Transfer Event Types** (`transfer.success`, `transfer.error`, `transfer.cancelled`): * `title`: A string describing the event, e.g. `"transfer.success for "` * `text`: The transfer log (truncated to 3000 characters if necessary) * `tags`: Array of tags, including: * `service:prequel` * `event_type:` * `transfer_id:` * `destination_id:` * `destination_name:` * `id_in_provider_system:` * `transfer_status:` * `rows_transferred:` * `env:`: available values are: `"staging"` and `"production"` * `host:`: the domain of the Prequel deployment that sent the event (e.g. `app.prequel.co`) * (optional) `transfer_blamed_party:`: available values are: `"first_party"`, `"third_party"`, `"prequel"`, and `"unknown"` * `date_happened`: Unix timestamp of the event * `alert_type`: `"info"` for success, `"error"` for error, `"user_update"` for cancelled * **Other Resource Event Types** (e.g. source, destination, recipient, magic link): * `title`: ` for ` * `text`: The event type, with dots replaced by spaces (e.g. `"recipient created"`) * `tags`: Array of tags, including: * `service:prequel` * `event_type:` * `resource_name:` * `env:`: available values are: `"staging"`, `"production"` * `host:`: the domain of the Prequel deployment that sent the event (e.g. `app.prequel.co`) * `date_happened`: Unix timestamp of the event * `alert_type`: `"user_update"` ## Example Datadog payloads **Transfer Success Example:** ```json title="Transfer success payload" icon="brackets-curly" expandable theme={null} { "title": "transfer.success for 00000000-0000-0000-0000-000000000000", "text": "mock_data - succeeded", "tags": [ "service:prequel", "event_type:transfer.success", "transfer_id:00000000-0000-0000-0000-000000000000", "destination_id:00000000-0000-0000-0000-000000000000", "destination_name:My Destination", "id_in_provider_system:dest_123", "transfer_status:SUCCESS", "rows_transferred:1000", "env:production", "host:app.prequel.co" ], "date_happened": 1708099148, "alert_type": "info" } ``` **Resource Event Example:** ```json title="Resource event payload" icon="brackets-curly" expandable theme={null} { "title": "recipient.created for Acme Inc", "text": "recipient created", "tags": [ "service:prequel", "event_type:recipient.created", "resource_name:Acme Inc", "env:production", "host:app.prequel.co" ], "date_happened": 1708099148, "alert_type": "user_update" } ``` ## Authentication The Datadog API key is required for Prequel to send events Datadog. You must provide this key when creating the webhook. # Logging Source: https://docs.prequel.co/export/monitoring/monitoring Events Prequel automatically logs, reviewable through the logs API endpoint. ## Monitoring Once Prequel is up and running, you may wish to monitor activity on the platform. Prequel automatically logs the following events which can be reviewed using the [logs API endpoint](/export/api-reference/monitoring-&-logging/list-logs). ### Resource management events | `event_type` | `resource_id` | Logged when... | | ------------------------ | ------------------ | ---------------------------- | | `DESTINATION_ADD` | `{destination_id}` | A destination is created | | `DESTINATION_DELETE` | `{destination_id}` | A destination is deleted | | `DESTINATION_UPDATE` | `{destination_id}` | A destination is updated | | `DESTINATION_ENABLE_ALL` | *(empty)* | All destinations are enabled | | `DESTINATION_PAUSE_ALL` | *(empty)* | All destinations are paused | | `SOURCE_ADD` | `{source_id}` | A source is created | | `SOURCE_DELETE` | `{source_id}` | A source is deleted | | `SOURCE_UPDATE` | `{source_id}` | A source is updated | | `RECIPIENT_ADD` | `{recipient_id}` | A recipient is created | | `RECIPIENT_DELETE` | `{recipient_id}` | A recipient is deleted | | `RECIPIENT_UPDATE` | `{recipient_id}` | A recipient is updated | | `MODEL_ADD` | `{model_id}` | A model is created | | `MODEL_UPDATE` | `{model_id}` | A model is updated | | `MODEL_DELETE` | `{model_id}` | A model is deleted | | `PRODUCT_ADD` | `{product_id}` | A product is created | | `PRODUCT_UPDATE` | `{product_id}` | A product is updated | | `PRODUCT_DELETE` | `{product_id}` | A product is deleted | ### Magic links & webhooks | `event_type` | `resource_id` | Logged when... | | ------------------- | ----------------- | ----------------------- | | `MAGIC_LINK_CREATE` | `{magic_link_id}` | A magic link is created | | `MAGIC_LINK_DELETE` | `{magic_link_id}` | A magic link is deleted | | `WEBHOOK_ADD` | `{webhook_id}` | A webhook is created | | `WEBHOOK_DELETE` | `{webhook_id}` | A webhook is deleted | | `WEBHOOK_UPDATE` | `{webhook_id}` | A webhook is updated | ### Transfer events Note: Due to volume of transfers, Prequel does not report individual transfers within logs. For more detail on transfer metrics, see [Usage Data](/export/features/usage-data). | `event_type` | `resource_id` | Logged when... | | ------------------- | --------------- | ------------------------------- | | `TRANSFER_CANCELED` | `{transfer_id}` | A transfer is manually canceled | ### Credential management | `event_type` | `resource_id` | Logged when... | | ------------------------------------ | ----------------- | ---------------------------------------- | | `RECIPIENT_SOURCE_CREDENTIAL_ADD` | `{credential_id}` | A recipient source credential is created | | `RECIPIENT_SOURCE_CREDENTIAL_DELETE` | `{credential_id}` | A recipient source credential is deleted | | `RECIPIENT_SOURCE_CREDENTIAL_UPDATE` | `{credential_id}` | A recipient source credential is updated | # Overview Source: https://docs.prequel.co/export/monitoring/overview Monitor your Prequel integration with structured logs and webhooks. Prequel gives you two complementary ways to observe your integration: **structured logs** of every resource change, and **webhooks** that push events to your systems as they happen. The events Prequel records (sources, destinations, recipients, models, and more), reviewable through the logs API. Subscribe to event types and deliver them over HTTPS, Slack, Datadog, or PagerDuty. # Slack Source: https://docs.prequel.co/export/monitoring/slack Use webhooks to monitor Prequel events directly in your Slack account. Use this guide as an example of how to setup your Slack account to receive Prequel webhook events. ## Create a Slack app in your Slack account 1. Create a new channel or find a channel where you'd like to deliver Prequel webhook events. 2. Navigate to the **[Create App](https://api.slack.com/apps/new)** flow in your browser and click "From scratch". 3. Create a name, for example **"Prequel Webhook Notification App"**, and pick your workspace where you'd like to send the events. Click **"Create App"**. ![](https://storage.googleapis.com/prequel_docs/images/slack-app-create.png) 1. In the **Features and functionality** section, select **Incoming webhooks**. Turn "**Activate Incoming Webhooks**" to the "**On**" position. At the bottom of the menu, under "Generate a Webhook URL for your Workspace", click "**Add New Webhook to Workspace**". In the following prompt, select the channel where you'd like to receive notifications and click "**Allow**". 2. Record the generated URL. It should look like `https://hooks.slack.com/services/W098X7YZZ6/A12BC34DEFG/AbCdEfgHIjklM12no3PqrSTU`. Learning about advanced Slack App functionality For more information about configuring your Slack App, you can refer to Slack's documentation: [https://api.slack.com/messaging/webhooks](https://api.slack.com/messaging/webhooks) ## Add your Slack web request URL as your webhook target 1. Using your **Prequel API key** and **Webhook URL**, subscribe to the events you'd like to post to your selected Slack channel destination. You can find the API documentation here (or `POST` the request using the interactive request builder): [Create webhook](/export/api-reference/webhooks/create-webhook) ```bash title="Create webhook request" icon="terminal" expandable theme={null} curl --request POST \ --url https:///webhooks \ --header 'X-API-KEY: ' \ --header 'content-type: application/json' \ --data ' { "webhook": { "vendor": "slack", "api_version": "2023-10-15", "name": "Slack Webhook Subscription", "url": "https://hooks.slack.com/services///", "events": [ "transfer.error", "destination.created" ] } } ' ``` ## FAQ Why does Prequel have a specific Slack webhook type? Slack recommends a specific payload shape for incoming webhooks. When `"vendor":"slack"` is used when creating the webhook subscription, Prequel will use a specific JSON payload that can be interpreted easily by Slack for every message. # Webhooks Source: https://docs.prequel.co/export/monitoring/webhooks Create endpoints to receive asynchronous events from your Prequel integration, such as transfer failures. You may wish to create endpoints to receive asynchronous events from your Prequel integration, such as transfer failures. Webhook endpoints created using the `POST /webhooks` [endpoint](/export/api-reference/webhooks/create-webhook) can subscribe to any of Prequel's [Event Types](/export/monitoring/webhooks#event-types) and configure delivery via HTTPS, Slack, or Pagerduty. Choose a delivery method to get started: Deliver events into a Slack channel for the team to triage. Forward events into Datadog as monitor-ready events. Send `HTTPS` POST or GET callbacks to your own receiver. For receiving sensitive trace alerts during transfers, see [Sensitive Trace Webhooks](/export/error-handling/sensitive-trace-webhook). ## Delivery methods ### HTTP POST and GET Prequel supports `HTTPS` callbacks to your custom webhook receiver. Creating a webhook endpoint as the `generic_post` type will cause payloads to be sent as a JSON payload. The `generic_get` type will deliver payloads as URL parameters. The payloads are defined in Event Types. ### PagerDuty, Slack & Datadog You may want to have events sent to specific vendors for special handling of webhook requests. Currently, PagerDuty, Slack and Datadog are supported, with specific payload shapes according to the vendor specifications. For detailed instructions on integrating Prequel webhooks with Slack or Datadog please visit the following pages: * [Slack Integration](/export/monitoring/slack) * [Datadog Integration](/export/monitoring/datadog) ## Authentication You can provide an API key for the webhook to use if the destination requires one. Additionally, Prequel signs every payload and passes the signature through the `X-Prequel-Webhook-Signature` header. See [Verifying Webhooks](/export/monitoring/webhooks#verifying-webhooks) for more information on handling the signature. ## Versioning Prequel uses a top-level `api_version` field. A version is represented by the date it was released. Currently, all webhooks use the version `2023-10-15`. ## Contents & structure All event types follow this structure: ### Headers | Header | Description | | :---------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Content-Type` | Always `application/json` | | `X-Prequel-Webhook-Timestamp` | Timestamp generated when the event is sent. | | `X-Prequel-Webhook-Signature` | Signature generated by Prequel using `SHA-256` and `RSA PKCS1 v1.5` signature scheme. See [Verifying Webhooks](/export/monitoring/webhooks#verifying-webhooks). | | `X-Prequel-Webhook-Digest` | Optional utility for signature verification. See [Verifying Webhooks](/export/monitoring/webhooks#verifying-webhooks). | ### Body ```json title="Body structure" icon="brackets-curly" expandable theme={null} { "type": "resource.event", "api_version": "XXXX-XX-XX", "created_at": ..., // timestamp generated when the event is created "data": { // event specific } } ``` ## Event types When creating a webhook, you must specify which events it should listen to. Webhooks created before October 31 2023 will continue to receive only transfer errors. Webhooks are available for the following event types: * `transfer.success` * `transfer.error` * Sent for transfers with the following [statuses](/export/concepts/transfers#transfer-lifecycle): `ERROR`, `PARTIAL_FAILURE`, `EXPIRED` * `transfer.cancelled` * `export_source.created` * `export_source.updated` * `export_source.deleted` * `export_destination.created` * `export_destination.updated` * `export_destination.deleted` * `recipient.created` * `recipient.updated` * `recipient.deleted` * `export_magic_link.created` * `export_magic_link.deleted` Each event type's payload is shown below. Select a tab to see the example for a given event. ### Transfer events This event has type `transfer.success`. It is sent whenever Prequel identifies a successful transfer. ```json title="transfer.success payload" icon="brackets-curly" expandable theme={null} { "type":"transfer.success", "api_version": "2023-10-15", "created_at": "2023-10-15T19:19:08+00:00", "data":{ "destination_id":"00000000-0000-0000-0000-000000000000", "destination_name":"Foo Bar", "id_in_provider_system":"acme", "in_app_url":"https://app.prequel.co/destinations/00000000-0000-0000-0000-000000000000", "is_full_refresh":true, "models":["users"], "rows_transferred":1000, "transfer_id":"00000000-0000-0000-0000-000000000000", "transfer_log":"users - succeeded", "transfer_log_pretty":"users - succeeded", "transfer_started_at": "2023-10-15T16:19:08+00:00", "transfer_ended_at": "2023-10-15T18:19:08+00:00", "transfer_status":"SUCCESS" }, } ``` This event has type `transfer.error`. It is sent whenever a transfer ends in a failure status: `ERROR`, `PARTIAL_FAILURE`, or `EXPIRED`. The `transfer_status` field in the payload identifies which failure occurred. Note the following fields which do not appear on the transfer object itself: * `transfer_blamed_party`: May be any of `first_party`, `third_party`, `prequel`, `host`, or `unknown`. Defaults to `prequel`. `first_party` indicates an error with your Prequel integration. `third_party` indicates an error caused by your customer; for example, a change in existing credentials for a destination. `host` is only applicable if you are self-hosting Prequel. For `EXPIRED` transfers, this field is always `prequel`. * `transfer_log`: Per-model breakdown of the transfer result. Each line follows the format `{model_name} - {status}`, where status is one of `succeeded`, `skipped, no updates`, or a raw error message including a `PRQL-XXXX` coded error code. * `transfer_log_pretty`: If `transfer_blamed_party` is `third_party`, a modified error string suitable for displaying to customers. Otherwise empty. ```json title="transfer.error payload" icon="brackets-curly" expandable theme={null} { "type": "transfer.error", "api_version": "2023-10-15", "created_at": "2023-10-15T19:19:08+00:00", "data": { "destination_id": "00000000-0000-0000-0000-000000000000", "destination_name": "Foo Bar", "id_in_provider_system": "acme", "models": ["users", "orders", "products", "reports"], "rows_transferred": 1179, "transfer_blamed_party": "third_party", "is_full_refresh": true, "transfer_id": "00000000-0000-0000-0000-000000000000", "transfer_status": "PARTIAL_FAILURE", "transfer_log": "users - succeeded\norders - succeeded\nproducts - skipped, no updates\nreports - unable to execute operation \"athena.source.read\": unable to read athena source: unable to create athena source query view: Driver Error: unable to execute create view query: Query execution timed out [PRQL-XIEA]: coded error message: The Athena query execution timed out. If this error persists, please contact support.", "transfer_log_pretty": "users - succeeded\norders - succeeded\nproducts - skipped, no updates\nreports - The Athena query execution timed out. If this error persists, please contact support.", "transfer_started_at": "2023-10-15T16:19:08+00:00", "transfer_ended_at": "2023-10-15T18:19:08+00:00", "in_app_url": "https://app.prequel.co/destinations/00000000-0000-0000-0000-000000000000" } } ``` This event has type `transfer.cancelled`. It is sent whenever Prequel identifies a cancelled or killed transfer. A transfer could be cancelled by the user when it is in the `PENDING` state, or it could be killed by the system or user if it is in the `RUNNING` state. Both actions will trigger a `transfer.cancelled` event. ```json title="transfer.cancelled payload" icon="brackets-curly" expandable theme={null} { "type": "transfer.cancelled", "api_version": "2023-10-15", "created_at": "2025-02-20T20:54:51.971902Z", "data": { "destination_id": "00000000-0000-0000-0000-000000000000", "destination_name": "Foo Bar", "id_in_provider_system": "acme", "in_app_url": "https://app.prequel.co/destinations/00000000-0000-0000-0000-000000000000", "is_full_refresh": false, "models": ["users"], "rows_transferred": 0, "transfer_ended_at": "2025-02-20T20:54:51.971902Z", "transfer_id": "00000000-0000-0000-0000-000000000000", "transfer_log": "Pending transfer cancelled by user.", "transfer_log_pretty": "", "transfer_started_at": null, "transfer_status": "CANCELLED" } } ``` ### Source events This event has type `export_source.created`. It is sent whenever Prequel identifies that a new source has been created. This example shows a new Snowflake source. ```json title="export_source.created payload" icon="brackets-curly" expandable theme={null} { "type": "export_source.created", "api_version": "2023-10-15", "created_at": "2025-02-20T17:54:18.453379Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "vendor": "snowflake", "database": "source_database", "username": "snowflake_user", "created_at": "2025-02-20T12:54:18.445223-05:00", "updated_at": "2025-02-20T12:54:18.445223-05:00", "disable_ssl": false, "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 22, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "max_concurrent_queries_per_transfer": 1 }, "previous_resource": null } } ``` This event has type `export_source.updated`. It is sent whenever Prequel identifies that an existing source has been updated. In this example, you can see the `database` field has been updated to `source_db_updated`. ```json title="export_source.updated payload" icon="brackets-curly" expandable theme={null} { "type": "export_source.updated", "api_version": "2023-10-15", "created_at": "2025-02-20T17:57:04.201273Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "vendor": "snowflake", "database": "source_db_updated", "username": "snowflake_user", "created_at": "2025-02-20T12:54:18.445223-05:00", "updated_at": "2025-02-20T12:57:04.195446-05:00", "disable_ssl": false, "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 22, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "max_concurrent_queries_per_transfer": 1 }, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "vendor": "snowflake", "database": "source_db", "username": "snowflake_user", "created_at": "2025-02-20T12:54:18.445223-05:00", "updated_at": "2025-02-20T12:54:18.445223-05:00", "disable_ssl": false, "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 22, "bucket_secret_key": "[redacted]", "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "max_concurrent_queries_per_transfer": 1 } } } ``` This event has type `export_source.deleted`. It is sent whenever Prequel identifies that an existing source has been deleted. ```json title="export_source.deleted payload" icon="brackets-curly" expandable theme={null} { "type": "export_source.deleted", "api_version": "2023-10-15", "created_at": "2025-02-20T17:59:09.416441Z", "data": { "current_resource": null, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "vendor": "snowflake", "database": "source_db_updated", "username": "snowflake_user", "created_at": "2025-02-20T12:54:18.445223-05:00", "updated_at": "2025-02-20T12:57:04.195446-05:00", "disable_ssl": false, "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 22, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" } } } } ``` ### Destination events This event has type `export_destination.created`. It is sent whenever Prequel identifies that a new destination has been created. ```json title="export_destination.created payload" icon="brackets-curly" expandable theme={null} { "type": "export_destination.created", "api_version": "2023-10-15", "created_at": "2025-02-20T18:12:45.409267Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "schema": "webhook_schema", "vendor": "snowflake", "database": "destination_db", "products": ["billing", "invoices"], "username": "snowflake_user", "created_at": "2025-02-20T13:12:45.360283-05:00", "is_enabled": true, "updated_at": "2025-02-20T13:12:45.360283-05:00", "disable_ssl": false, "recipient_id": "00000000-0000-0000-0000-000000000000", "enabled_models": ["*"], "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 0, "frequency_minutes": 0, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "id_in_provider_system": "tenant_id", "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "last_successful_transfer_ended_at": null, "max_concurrent_queries_per_transfer": 1 }, "previous_resource": null } } ``` This event has type `export_destination.updated`. It is sent whenever Prequel identifies that an existing destination has been updated. ```json title="export_destination.updated payload" icon="brackets-curly" expandable theme={null} { "type": "export_destination.updated", "api_version": "2023-10-15", "created_at": "2025-02-20T18:15:11.411273Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "schema": "new_webhook_schema", "vendor": "snowflake", "database": "destination_db", "products": ["billing", "invoices"], "username": "snowflake_user", "created_at": "2025-02-20T13:12:45.360283-05:00", "is_enabled": true, "updated_at": "2025-02-20T13:15:11.398688-05:00", "disable_ssl": false, "recipient_id": "00000000-0000-0000-0000-000000000000", "enabled_models": ["*"], "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 0, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "id_in_provider_system": "tenant_id", "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "last_successful_transfer_ended_at": null, "max_concurrent_queries_per_transfer": 1 }, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "schema": "webhook_schema", "vendor": "snowflake", "database": "destination_db", "products": ["billing", "invoices"], "username": "snowflake_user", "created_at": "2025-02-20T13:12:45.360283-05:00", "is_enabled": true, "updated_at": "2025-02-20T13:12:45.360283-05:00", "disable_ssl": false, "recipient_id": "00000000-0000-0000-0000-000000000000", "enabled_models": ["*"], "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 0, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "id_in_provider_system": "tenant_id", "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "last_successful_transfer_ended_at": null, "max_concurrent_queries_per_transfer": 1 } } } ``` This event has type `export_destination.deleted`. It is sent whenever Prequel identifies that an existing destination has been deleted. ```json title="export_destination.deleted payload" icon="brackets-curly" expandable theme={null} { "type": "export_destination.deleted", "api_version": "2023-10-15", "created_at": "2025-02-20T18:17:31.963946Z", "data": { "current_resource": null, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "name": "Acme Inc", "port": 443, "schema": "new_webhook_schema", "vendor": "snowflake", "database": "destination_db", "products": ["billing", "invoices"], "username": "snowflake_user", "created_at": "2025-02-20T13:12:45.360283-05:00", "is_enabled": true, "updated_at": "2025-02-20T13:15:11.398688-05:00", "disable_ssl": false, "recipient_id": "00000000-0000-0000-0000-000000000000", "enabled_models": ["*"], "use_ssh_tunnel": false, "ssh_tunnel_host": "", "ssh_tunnel_port": 0, "ssh_tunnel_username": "", "gcp_iam_role_metadata": { "type": "", "audience": "", "token_url": "", "credential_source": { "url": "", "region_url": "", "environment_id": "", "regional_cred_verification_url": "" }, "subject_token_type": "", "service_account_impersonation_url": "" }, "id_in_provider_system": "tenant_id", "max_concurrent_transfers": 1, "is_bucket_credentials_implicit": false, "last_successful_transfer_ended_at": null, "max_concurrent_queries_per_transfer": 1 } } } ``` ### Recipient events This event has type `recipient.created`. It is sent whenever Prequel identifies that a new recipient has been created. ```json title="recipient.created payload" icon="brackets-curly" expandable theme={null} { "type": "recipient.created", "api_version": "2023-10-15", "created_at": "2025-02-20T17:44:10.769913Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "name": "Acme Inc", "id_in_provider_system": "acme_inc", "products": ["billing", "invoices"], "created_at": "2025-02-20T12:44:10.763224-05:00", "updated_at": "2025-02-20T12:44:10.763224-05:00", }, "previous_resource": null } } ``` This event has type `recipient.updated`. It is sent whenever Prequel identifies that an existing recipient has been updated. In this example, you can see that the `products` field has been updated to include `invoices`. ```json title="recipient.updated payload" icon="brackets-curly" expandable theme={null} { "type": "recipient.updated", "api_version": "2023-10-15", "created_at": "2025-02-20T17:45:56.752967Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "name": "Acme Inc", "id_in_provider_system": "acme_inc", "products": ["billing", "invoices"], "created_at": "2025-02-20T12:44:10.763224-05:00", "updated_at": "2025-02-20T12:45:56.746663-05:00", }, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "name": "Acme Inc", "id_in_provider_system": "acme_inc", "products": ["billing"], "created_at": "2025-02-20T12:44:10.763224-05:00", "updated_at": "2025-02-20T12:44:10.763224-05:00", } } } ``` This event has type `recipient.deleted`. It is sent whenever Prequel identifies that an existing recipient has been deleted. ```json title="recipient.deleted payload" icon="brackets-curly" expandable theme={null} { "type": "recipient.deleted", "api_version": "2023-10-15", "created_at": "2025-02-20T17:48:35.128964Z", "data": { "current_resource": null, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "name": "Acme Inc", "id_in_provider_system": "acme_inc", "products": ["billing", "invoices"], "created_at": "2025-02-20T12:44:10.763224-05:00", "updated_at": "2025-02-20T12:45:56.746663-05:00" } } } ``` ### Magic link events This event has type `export_magic_link.created`. It is sent whenever Prequel identifies that a new magic link has been created. ```json title="export_magic_link.created payload" icon="brackets-curly" expandable theme={null} { "type": "export_magic_link.created", "api_version": "2023-10-15", "created_at": "2025-02-20T21:13:53.155964Z", "data": { "current_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "link": "https://app.prequel.co/orgs/00000000-0000-0000-0000-000000000000/links/00000000-0000-0000-0000-000000000000", "name": "Webhook Magic Link Acme Inc", "vendor": "snowflake", "products": ["billing", "invoices"], "created_at": "2025-02-20T16:13:53.149884-05:00", "bucket_name": "", "recipient_id": "00000000-0000-0000-0000-000000000000", "has_been_used": false, "ssh_public_key": "ssh-rsa...prequel-ssh-tunneling-public-key", "available_models": ["Customers", "Invoices"], "set_destination_enabled": true }, "previous_resource": null } } ``` This event has type `export_magic_link.deleted`. It is sent whenever Prequel identifies that an existing magic link has been deleted. ```json title="export_magic_link.deleted payload" icon="brackets-curly" expandable theme={null} { "type": "export_magic_link.deleted", "api_version": "2023-10-15", "created_at": "2025-02-20T21:16:58.035652Z", "data": { "current_resource": null, "previous_resource": { "id": "00000000-0000-0000-0000-000000000000", "host": "snowflake-webhook.region.cloud.snowflakecomputing.com", "link": "https://app.prequel.co/orgs/00000000-0000-0000-0000-000000000000/links/00000000-0000-0000-0000-000000000000", "name": "Webhook Magic Link Acme Inc", "vendor": "snowflake", "products": ["billing", "invoices"], "created_at": "2025-02-20T16:13:53.149884-05:00", "bucket_name": "", "recipient_id": "00000000-0000-0000-0000-000000000000", "has_been_used": false, "ssh_public_key": "ssh-rsa...prequel-ssh-tunneling-public-key", "available_models": ["Customers", "Invoices"], "id_in_provider_system": "acme", "set_destination_enabled": true } } } ``` ## Verifying webhooks Prequel provides a unique signature in the HTTP header of each webhook request. You can use this signature and your account's webhook public key to verify that the data you receive is from Prequel. ### Webhook signatures Prequel's approach to webhook signatures is based on asymmetric cryptography. Prequel generates an [RSA](https://en.wikipedia.org/wiki/RSA_\(cryptosystem\)) private and public key pair for your account's webhook integration. After generating a webhook, Prequel digitally signs the payload with the private key. On receiving the webhook, you can use the public key to verify the authenticity and freshness of this signature. See [Verify the signature](/export/monitoring/webhooks#verify-the-signature). ### Relevant headers | Header | Description | | :--------------------------------- | :---------------------------------------------------------------------------------------- | | `X-Prequel-Webhook-Timestamp` | Timestamp the webhook was sent, in `RFC 3339` format. | | `X-Prequel-Webhook-Signature` | Hex-encoded signature generated by Prequel using the signing data. | | (Maybe) `X-Prequel-Webhook-Digest` | Hex-encoded SHA-256 hash of the payload only, provided by Prequel for debugging purposes. | ### Verify the signature The general steps to verify a signature are outlined below. #### 1. Retrieve your current webhook key You can retrieve your current webhook key by hitting the following API endpoint `/public/signatures/webhook-public-key`. #### 2. Reconstruct the signing data Extract the timestamp provided in the `X-Prequel-Webhook-Timestamp` header. The date is in RFC 3339 format and looks something like: `X-Prequel-Webhook-Timestamp: 2023-10-15T14:30:00Z` The signing data is in the format `timestamp.body`, constructed by concatenating * The timestamp * The character `.` * The request body, i.e. the raw JSON payload Finally, hash the **full signing data** using `SHA-256`. Prequel uses the raw body of the request to generate a signature. You should hash the raw body string (or bytes) before deserializing the JSON payload. Frameworks that parse the response body may introduce subtle changes, such as removing characters or changing key-sort order. ##### Validate the response hash Prequel hashes the raw body only and provides the result in the `X-Prequel-Webhook-Digest` header. You can use this to validate that you are handling the response body correctly by hex-decoding the value and comparing it to your SHA-256 hash of the raw body. **You should not use this to confirm the signature.** #### 3. Confirm the signature Once you've reconstructed and hashed the signing data, you can sign the data with your public key and compare the output to the signature in the `X-Prequel-Webhook-Signature` header. Make sure to decode Prequel's signature from hex before comparing. Prequel uses the PKCS1 v1.5 signature scheme. #### 4. Verify timing A replay attack is when an attacker intercepts a valid payload and its signature, then re-transmits them. The provided timestamp is verified by the signature, so an attacker can't change the timestamp without invalidating the signature. However, even if the attacker cannot alter the data, they may produce side-effects from processing the same event multiple times. We recommend defining a time window (such as 5 minutes) and rejecting events that are too old. # Authentication Source: https://docs.prequel.co/export/sdks/react/authentication How to authenticate Prequel React SDK requests with API keys and scoped auth tokens. ## Client vs. server authentication ### Server authentication Every Prequel account has access to associated API keys. These API keys can be used to replicate any behavior you see in the Prequel Admin UI. ### Client authentication If you decide to embed your Prequel powered features in your app for your users to self-serve, you may use your API keys to generate **Scoped Auth Tokens** for use in your clients. ## Overview of authentication keys/tokens **Understanding Our Terminology: First Party vs. Third Party** * **First Party** refers to Prequel users (the First Party). * **Third Party** refers to Prequel users' (the First Party's) customers. | Type of Prequel Authentication Token | Intended Usage | Scope | | :----------------------------------- | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **API keys** | Server | Scoped to a specific First Party **Staging** or **Production** Environment (including all Third Party resources) | | **Scoped Auth Token**: *POST* | Client | Scoped to a specific Third Party & destination. Grants the ability to POST resources with only those destination details. Grants the ability to PATCH resources with only those destination details, except for host and bucket name which are prohibited to PATCH for security reasons. | | **Scoped Auth Token**: *GET* | Client | Scoped to a specific Third Party. Grants the ability to GET all related Third Party resources and details. | ## Generating scoped auth tokens The generation of a **Scoped Auth Token** (required for all Prequel React SDK Hooks) must happen on the server. This short-lived **Scoped Auth Token** is used to authenticate client-side requests made to Prequel. This scoped token allows you to make requests directly from your web application to Prequel, without having to proxy them through the backend. To generate this auth token, your backend should make an API request to Prequel's [Generate Scoped Auth Token endpoint](/export/api-reference/auth/generate-scoped-auth-token). It's important that the backend specify certain fields in the destination passed to this request: specifically, the user's `id_in_provider_system`. This is how Prequel knows that this token can only be used to submit and retrieve data for a given tenant. Once you've implemented the backend functionality to generate the token, you will need to define a `fetchToken` function in your frontend that hits the relevant backend endpoint, and that can be passed to Prequel hooks. ![](https://storage.googleapis.com/prequel_docs/images/react-example-scoped-auth-guide.png) # Hooks Source: https://docs.prequel.co/export/sdks/react/hooks A comprehensive set of React Hooks to embed Prequel functionality into your React app. The Prequel React SDK provides a comprehensive set of React Hooks to help you embed the functionality of Prequel into your React app. For a full list of available hooks, helper functions, and examples, [see our React SDK documentation](https://www.npmjs.com/package/@prequel/react). ## Hooks vs. components Prequel provides a comprehensive set of React Hooks to help you embed Prequel's functionality into your app, though there are no pre-built React Components at this time. We hope the provided Hooks can help you build your own React Components into your app that adhere perfectly to your branding and design guides. # Overview Source: https://docs.prequel.co/export/sdks/react/overview The Prequel React SDK is a wrapper around the Prequel API for embedding customer-facing data syncing in your React app. Our React SDK, `Prequel/React`, is a wrapper around our Prequel API. It allows you embed your customer-facing data syncing feature to your React webapp. ## Example of an "add destination" form built using `Prequel/React` ![](https://storage.googleapis.com/prequel_docs/images/react-example-add-destination-form.png "Example of an \"Add Destination\" form powered by Prequel's React SDK.") ## Setup Install Prequel React from the [`npm` public registry](https://www.npmjs.com/package/@prequel/react). ```bash title="Command Line" icon="terminal" theme={null} npm install --save @prequel/react ``` ## Example React app Checkout our [React Example App](https://github.com/prequel-co/react-example) for an in-depth look on how to build a dynamic form with our SDK. ## Explore the React SDK Generate scoped auth tokens to authenticate client-side requests. The set of React Hooks for embedding Prequel into your app. A barebones example app to help you get started. ## Tutorials Step-by-step guides for building common destination flows: * [Add Destination Form](/export/sdks/react/tutorials/building-an-add-destination-connection-form): build a dynamic form to test and connect destinations. * [Edit Destination Form](/export/sdks/react/tutorials/building-an-edit-destination-connection-form): build a form to edit, test, and update destinations. * [Viewing Existing Destinations](/export/sdks/react/tutorials/viewing-existing-destinations): render existing destinations for your users to inspect. * [Viewing Destination Transfers](/export/sdks/react/tutorials/viewing-destination-transfers): render transfers for an existing destination. * [Deleting Destinations](/export/sdks/react/tutorials/deleting-destinations): build functionality for your users to delete destinations. # Example app Source: https://docs.prequel.co/export/sdks/react/sample-app We've built a barebones "Sample App" to help you get started. ## Explore the example app For a more in-depth look at how to use the React SDK, check out our React Example App [here](https://github.com/prequel-co/react-example). # Building an "Add Destination" connection form Source: https://docs.prequel.co/export/sdks/react/tutorials/building-an-add-destination-connection-form Build a dynamic "Add Destination" form to allow your users to test and connect their destinations. In your backend, build an API endpoint that makes a `POST` to `https://api.prequel.co/actions/generate-scoped-auth-token` to request to generate a scoped auth token from the Prequel API. Be sure to pass your API Key as a header (`X-API-KEY`) in the request as well as the desired API version (`X-Prequel-Api-Version`). Your API endpoint should return the scoped token that was returned from the Prequel API. *Note: If you are using a self-hosted deployment or a non-US deployment, replace `api.prequel.co` with the appropriate host.* ### POST request body This request body is for the API Version `2023-12-01`. Your `POST` request body should have the following structure: | Parameter | Type | Notes | | -------------------- | :------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `recipient_id` | `string` | The Recipient ID for your customer. This value should be derived in your backend from a server-side token or your preferred method to associate user sessions with Recipient IDs. | | `destination` | `Object` | This is the `Destination` object sent from the Prequel React hooks. You can perform any validation on the object that you wish and send it on to the Prequel API. | | `application_origin` | `string` | The application host for your React app that will be making requests to the Prequel API using the generated scoped auth token. | ##### Example request body ```json title="Example request body" icon="brackets-curly" expandable theme={null} { "recipient_id": "1abc234d-efg5-6h7i-j8k9-012345lm6nop", "destination": , "application_origin": "app.example.co" } ``` ### Getting the recipient ID The `recipient_id` is Prequel's unique identifier for the recipient to which the destination is being added. In your system, there is a separate identifier for that customer. You can either you can look them up using the Prequel API or track Prequel's recipient UUIDs in your system. ### POST response body The `POST` response body will have the following structure: | Parameter | Type | Notes | | -------------- | :------- | :-------------------------------------------------------------------------------------------- | | `scoped_token` | `string` | The scoped auth token generated for the specific destination context supplied in the request. | ##### Example response body ```json title="Example response body" icon="brackets-curly" expandable theme={null} { "scoped_token": "eyJhb..." } ``` Note: scoped auth tokens have a TTL of one hour. In your frontend, define a set of `fetchToken` functions. These functions will consume various inputs (see examples below) and query the API endpoint you built in Step 1 to generate auth tokens required by the Prequel API. These `fetchToken` functions will be used as parameters to and called by Prequel's React hooks. There are two `fetchToken` types you will need to implement, depending on the hooks you use: ```typescript title="fetchToken types" icon="code" theme={null} export type FetchAuthToken = () => Promise; export type FetchAuthTokenWithDestination = (d: PreparedDestination) => Promise; ``` ### Example: `FetchAuthToken` Required by: `useModels`, `useProducts` hooks This function contains an empty request body (as should your subsequent request to the Prequel API) because there is no relevant customer-specific scoping required for the fetching of your organization's models and products. ```typescript title="fetchToken" icon="code" expandable theme={null} const fetchToken: (d: PreparedDestination) => Promise = (destination) => fetch(`https://api.example.co/generate-auth-token`, { method: "POST", headers: { "Content-Type": "application/json" }, body: {}, }) .then((response: Response) => response.json()) .then((body) => body.scoped_token) .catch((reason) => { console.error(reason); }); ``` ### Example: `FetchAuthTokenWithDestination` Required by: `useCreateDestination`, `useTestConnection` hooks This function passes a destination object as the request body (in addition to - not in place of - to subsequent requests to the Prequel API) so that the generated token can be appropriately scoped to a specific destination. ```typescript title="fetchTokenWithDestination" icon="code" expandable theme={null} const fetchTokenWithDestination: (d: PreparedDestination) => Promise = (destination) => fetch(`https://api.example.co/generate-auth-token`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(destination), }) .then((response: Response) => response.json()) .then((body) => body.scoped_token) .catch((reason) => { console.error(reason); }); ``` Install the `@prequel/react` package from [npm](https://www.npmjs.com/package/@prequel/react). ```bash title="Install @prequel/react" icon="terminal" theme={null} npm install @prequel/react@latest ``` In your frontend, define a `DestinationForm` component. The component will leverage the `useDestination` hook and the `useDestinationForm` hook to render a dynamic form that can be used to test & add a new destination. A brief example is show below. See [our sample React app](https://github.com/prequel-co/react-example/blob/main/src/DestinationForm.tsx) for a more comprehensive implementation. ```tsx title="DestinationForm" icon="react" expandable theme={null} const DestinationForm = () => { const formRef = useRef(null); const [destination, setDestination] = useDestination({ name: "Customer A Snowflake Destination", // Readable name for the destination }); const destinationForm = useDestinationForm( destination, PREQUEL_ORG_ID, // the UUID for your organization in Prequel { includeInternalFields: true, // Optional (default false): hide any non-user facing fields like "name" host: PREQUEL_HOST, // Optional (default api.prequel.co): the host url of your Prequel API vendorQuery: "snowflake" // Optional (default undefined): vendor to filter the form to } ); const createDestination = useCreateDestination( fetchTokenWithDestination, // the fetchToken function you created in step 2 "app.example.co", // the origin of your React app (used to allow CORS) PREQUEL_HOST // Optional (default api.prequel.co): the host url of your Prequel API ); // This is the PreparedDestination you can pass to testConnection and createDestination const preparedDestination = useMemo( () => prepareDestinationWithForm(destination, destinationForm), [destination, destinationForm] ); if (!destinationForm) { return (
Loading...
); } return (
{destinationForm.map((section) => ( {/* Render the section title and subtitle... */} {section.fields.map((field) => { {/* Render the field... */} })} ))} ); } ``` Once your form is rendering as desired, you should augment form functionality to test a destination connection before creating that destination. It's recommended that you only allow users to create destinations that are connecting successfully. You can leverage the `useTestConnection` hook as demonstrated below. See [our sample React app](https://github.com/prequel-co/react-example/blob/main/src/TestConnection.tsx) for a more comprehensive implementation. ```tsx title="TestConnectionComponent" icon="react" expandable theme={null} const TestConnectionComponent = ({ preparedDestination }) => { const [testRunning, setTestRunning] = useState(false); const [testResult, setTestResult] = useState(null); const testConnection = useTestConnection( fetchTokenWithDestination, // the fetchToken function you created in step 2 "app.example.co", // the origin of your React app (used to allow CORS) PREQUEL_HOST // the host url of your Prequel API ); async function testDestinationConnection() { setTestRunning(true); setTestResult("Testing new connection..."); const { data, message } = await testConnection(preparedDestination); if (data?.status === "success") { setTestResult("Connection test successful."); } else { setTestResult(message); } setTestRunning(false); } return (
{/* Render connection information as desired... */}
) } ``` # Building an "Edit Destination" connection form Source: https://docs.prequel.co/export/sdks/react/tutorials/building-an-edit-destination-connection-form Build a dynamic "Edit Destination" form to allow your users to edit, test, and update their destinations. Use the `useGetDestinationsForRecipient` hook to get a list of destinations and select the destination to edit. The example below uses the destination ID from the url to select the `currentDestination`, but there are many other viable solutions to obtain it. ```tsx title="Select destination to edit" icon="react" expandable theme={null} const destinationID = "X" // Obtain destination ID by your preference const [destinations, setDestinations] = useState(); const [currentDestination, setCurrentDestination] = useState(); const getDestinations = useGetDestinations( fetchToken, // fetchToken implementation, takes no parameters "app.example.co", // the origin of your React app (used to allow CORS), PREQUEL_HOST // Optional (default api.prequel.co): the host url of your Prequel API ); // Fetch destinations in useEffect elided... useEffect(() => { setCurrentDestination(destinations.find(({ id }) => id === destinationID); }, [destinations, destinationID]) ``` This step can be done by amending your "Create Destination" form to optionally accept a destination, or you can build a separate form implementation that strictly accepts a destination or destination ID. After selecting the destination to edit, populate the form with that existing destination's values. This can be done by leveraging the `prepareDestinationFromExisting` helper function and the `setDestination` function returned from the `useDestination` hook. ```tsx title="EditDestinationForm" icon="react" expandable theme={null} const EditDestinationForm = () => { // Destination selection code (above) elided const [destination, setDestination] = useDestination(); const destinationForm = useDestinationForm( destination, PREQUEL_ORG_ID, // the UUID for your organization in Prequel { includeInternalFields: true, // Optional (default false): hide any non-user facing fields like "name" host: PREQUEL_HOST, // Optional (default api.prequel.co): the host url of your Prequel API ); const updateDestination = useUpdateDestination( fetchTokenWithDestination, // fetchToken implementation, takes an ExistingDestination "app.example.co", // the origin of your React app (used to allow CORS) PREQUEL_HOST // the host url of your Prequel API ); useEffect(() => { if (currentDestination) { const converted = prepareDestinationFromExisting(currentDestination); setDestination(converted); // This line is what will populate the existing values in the destination } }, [setDestination, currentDestination]); // This is the PreparedDestination you can pass to testExistingConnection and updateDestination const preparedDestination = useMemo( () => prepareDestinationWithForm(destination, destinationForm), [destination, destinationForm] ); // If desired, include a loading spinner if values are undefined return (
{destinationForm.map((section) => ( {/* Render the section title and subtitle... */} {section.fields.map((field) => { {/* Render the field with destination value... */} })} ))} ); } ``` ### Non-required secret fields `ExistingDestination` objects fetched from Prequel will not include secret values (e.g. passwords, service account keys) for security reasons. When populating the form with existing values, those fields will remain empty. Though the `Form` object has them labeled as `required`, you should set those fields as optional in the form and only send *updated* secret values to the test and edit functions. If you do not include a secret value, Prequel will look up the destinations existing secret value when testing the connection. Once your form is properly populating and updating values on the destination, you should add functionality to test the destination. It's recommended that you only allow users to update destinations with values that are connecting successfully. You can leverage the `useTestExistingConnection` hook to add this functionality. ```tsx title="TestConnectionComponent" icon="react" expandable theme={null} const TestConnectionComponent = ({ destination, preparedDestination }) => { const [testRunning, setTestRunning] = useState(false); const [testResult, setTestResult] = useState(null); const testExistingConnection = useTestExistingConnection( fetchTokenWithDestination, // fetchToken implementation, takes an ExistingDestination "app.example.co", // the origin of your React app (used to allow CORS) PREQUEL_HOST // the host url of your Prequel API ); async function testDestinationConnection() { setTestRunning(true); setTestResult("Testing new connection..."); const { data, message } = await testExistingConnection(destination, preparedDestination); if (data?.status === "success") { setTestResult("Connection test successful."); } else { setTestResult(message); } setTestRunning(false); } return (
{/* Render connection information as desired... */}
) } ``` # Deleting destinations Source: https://docs.prequel.co/export/sdks/react/tutorials/deleting-destinations Build functionality for your users to delete their destinations. In your frontend, call the `useDeleteDestination` hook to generate a function that fetches a scoped auth token and uses it to delete the destination scoped by the token. Use one of the `fetchToken` functions you wrote in [Building an "Add Destination" connection form](/export/sdks/react/tutorials/building-an-add-destination-connection-form) here to create one. ```tsx title="Create deleteDestination" icon="react" expandable theme={null} const deleteDestination = useDeleteDestination( fetchToken, // fetchToken implementation, takes an ExistingDestination "app.example.co", // the origin of your React app (used to allow CORS), PREQUEL_HOST // Optional (default api.prequel.co): the host url of your Prequel API ); ``` Use `deleteDestination` created in Step 1 to delete the given destination. ```tsx title="DestinationView" icon="react" expandable theme={null} const DestinationView = () => { // Deletion function creation elided // Async function deletes destination using Prequel API async function onDelete(existing: ExistingDestination) => { const response = await deleteDestination(existing); if (response.status === "success") { // Handle success } else { // Handle failure } }; return ( ); } ``` # Viewing destination transfers Source: https://docs.prequel.co/export/sdks/react/tutorials/viewing-destination-transfers Render transfers for an existing destination for your users to inspect. In your frontend, call the `useGetTransfers` hook to generate a function that fetches a scoped auth token and uses it to get transfers for the destination scoped by the token. Use one of the `fetchToken` functions you wrote in [Building an "Add Destination" connection form](/export/sdks/react/tutorials/building-an-add-destination-connection-form) here to create one. ```tsx title="Create getTransfers" icon="react" expandable theme={null} const getTransfers = useGetTransfers( fetchToken, // fetchToken implementation, takes an ExistingDestination "app.example.co", // the origin of your React app (used to allow CORS), PREQUEL_HOST // Optional (default api.prequel.co): the host url of your Prequel API ); ``` Use `getTransfers` created in Step 1 to fetch transfers for the given destination. ```tsx title="Fetch transfers" icon="react" expandable theme={null} // Local state to store transfers const [transfers, setTransfers] = useState(); // Async function fetches transfers from Prequel API const fetchTransfers = async (existing: ExistingDestination) => { const transfers = await getTransfers(existing); setTransfers(transfers); }; useEffect(() => { fetchTransfers(destination); // Fetch the transfers for selected destination }, [destination]); ``` Render the `transfers` fetched from the Prequel API for inspection. ```tsx title="Transfers" icon="react" expandable theme={null} const Transfers = () => { // Transfers fetching elided return ( <> {transfers.map((t) => (

{t.id}

{t.status}

{t.log}

// ...other transfer fields
))} ); } ``` # Viewing existing destinations Source: https://docs.prequel.co/export/sdks/react/tutorials/viewing-existing-destinations Render existing destinations for your users to inspect. In your frontend, call the `useGetDestinationsForRecipient` hook to generate a function that fetches a scoped auth token and uses it to get destinations for the recipient scoped by the token. Use one of the `fetchToken` functions you wrote in [Building an "Add Destination" connection form](/export/sdks/react/tutorials/building-an-add-destination-connection-form) here to create one. ```tsx title="Create getDestinations" icon="react" expandable theme={null} const getDestinations = useGetDestinations( fetchTokent, // fetchToken implementation, takes no parameters "app.example.co", // the origin of your React app (used to allow CORS), PREQUEL_HOST // Optional (default api.prequel.co): the host url of your Prequel API ); ``` Use `getDestinations` created in Step 1 to fetch destinations for the given recipient. ```tsx title="Fetch destinations" icon="react" expandable theme={null} // Local state to store existing destinations const [destinations, setDestinations] = useState(); // Async function fetches existing destinations from Prequel API const fetchDestinations = async (id: string) => { const destinations = await getDestinations(id); setDestinations(destinations); }; useEffect(() => { fetchDestinations(); // Fetch the destinations for active user's recipient }, []); ``` Render the `destinations` fetched from the Prequel API for inspection. ```tsx title="RecipientDestinations" icon="react" expandable theme={null} const RecipientDestinations = () => { // Recipient fetching elided return ( <> {destinations.map((d) => (

{d.id}

{d.name}

{d.vendor}

// ...other destination fields
))} ); } ``` # Overview Source: https://docs.prequel.co/export/security/overview How Prequel keeps your data secure: no stored data, end-to-end encryption, flexible deployment, and SOC-2 Type II compliance. ## No stored data Prequel reads data from the source and writes it to the destination using an ephemeral server. No data remains on the platform after a transfer. See [Safeguarding user data](/export/security/safeguarding-user-data) for detail. ## Fully encrypted Prequel encrypts your data in transit and at rest. We use modern cryptographic algorithms like `AES256-GCM` and follow key management best practices with strict user access control. ## Deployment and connection options Prequel includes the deployment and connection controls enterprises need. **Deployment options:** * Cloud hosted * Private-cloud deployment * Self-hosted deployment See [Deployment options](/export/deployment/cloud-hosted). **Connection modalities:** * Role-based access control * Username and password authentication * IP whitelisting * SSH tunneling See [Connection modalities](/export/features/connection-modalities). ## Compliance and testing ### SOC-2 Type II compliant Prequel is SOC-2 Type II compliant. The way we process and store client data is secure and protected based on standards set by the AICPA. ### White box penetration testing Prequel regularly undergoes white box penetration testing. Contact us to get a copy of the latest report. # Safeguarding user data Source: https://docs.prequel.co/export/security/safeguarding-user-data How Prequel handles, isolates, and avoids retaining the data it transfers. As a matter of security and compliance, Prequel does not store nor retain any of the data it transfers. Transferred data only lives within the ephemeral worker tasked with running a specific transfer for the duration of the transfer and up to 24hrs afterwards. These workers are sandboxed from each other; a dedicated worker is spun up for each transfer and wound down afterwards. In order to facilitate incremental transfers, Prequel does store the timestamp corresponding to the most recent `last_modified_at` value for each transfer run. We consider this to be safe metadata rather than user data. # Additional options Source: https://docs.prequel.co/export/sources/additional-options/additional-options Optional source connection configurations: SSH tunneling and staging buckets. Configure optional connectivity and staging settings for your Prequel sources. ## SSH tunneling Configure a bastion server to accept an SSH connection from Prequel. Configure an AWS bastion server to accept an SSH connection from Prequel. Configure a GCP bastion server to accept an SSH connection from Prequel. Configure an Azure bastion server to accept an SSH connection from Prequel. ## Staging buckets Set up a staging bucket for efficient data transfer. | Option | Docs | | :---------------------------------- | :----------------------------------------------------------------------------- | | S3 Staging Bucket | [link](/export/sources/additional-options/s3-staging-bucket) | | Google Cloud Storage Staging Bucket | [link](/export/sources/additional-options/google-cloud-storage-staging-bucket) | | Azure Blob Storage Staging Bucket | [link](/export/sources/additional-options/azure-blob-storage-staging-bucket) | ## Other options | Option | Docs | | :------------------------------- | :-------------------------------------------------------------------------- | | Databricks Service Principal PAT | [link](/export/sources/additional-options/databricks-service-principal-pat) | # Azure Blob Storage staging bucket Source: https://docs.prequel.co/export/sources/additional-options/azure-blob-storage-staging-bucket Set up an Azure Blob Storage staging bucket for efficient destination data transfer. Some sources or destinations without built-in staging resources require a staging bucket to efficiently transfer or ingest data. ### Create storage account 1. In the Azure portal, navigate to the **Storage accounts** service and click **+ Create**. 2. In the "Basics" tab of the "Create a storage account" form, fill in the required details. 3. In the "Advanced" settings, under "Security" make sure **Enable storage account key access** is turned on. You may turn off (deselect) "Allow enabling public access on containers". Under "Data Lake Storage Gen2", select **Enable hierarchical namespace**. ![](https://storage.googleapis.com/prequel_docs/images/azure-settings-toggles.png) 4. In the "Networking" settings, you may limit "Network access" to **Enable public access from select virtual networks and IP addresses**. All other settings can use the default selections. 5. In the "Data protection" settings, you must turn off **Enable soft delete for blobs**, **Enable soft delete for containers**, and **Enable soft delete for file shares**. ![](https://storage.googleapis.com/prequel_docs/images/azure-turn-off-settings.png) 6. Once the remaining options have been configured to your preference, click **Create**. ### Create bucket and access token 1. In the Azure portal, navigate to the **Storage accounts** service and click on the account that was created in the previous step. 2. In the navigation pane, under "Data storage", click **Containers**. Click **+ Container**, choose a name for the container, and click **Create**. 3. In the navigation pane, under "Security + networking", click **Shared access signature**. 4. In the "Allowed services" list, select **Blob** and **File**. In the "Allowed resource types" list, select **Container** and **Object**. In the "Allowed permissions" list, select **Read**, **Write**, **Delete**, **List**, **Add**, **Create**, and **Permanently Delete**. 5. Select a "Start and expiry date/time" based on your security posture, and click **Generate SAS and connection string**. 6. Make a note of the **SAS token** that is generated. ![](https://storage.googleapis.com/prequel_docs/images/azure-setup-details.png) **Optional: Add a short retention lifecycle policy** You may configure a lifecycle management rule on the container to automatically delete blobs older than 2 days as the container is not used to persist data. In the storage account's **Lifecycle management** settings, add a rule to delete blobs after 2 days. Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ### You're done! Use this configured Azure Blob Storage staging bucket during the connection of your preferred data destination. # Azure SSH tunneling Source: https://docs.prequel.co/export/sources/additional-options/azure-ssh-tunneling Configure an Azure bastion server to accept an SSH connection from Prequel. Follow these steps to configure an Azure bastion server to accept an SSH connection from Prequel. ### Prerequisites * A public SSH key to add to the bastion machine to authenticate the SSH connection. *Ask your contact at Prequel if they have not already provided one to you.* ### Selecting your bastion server If using SSH tunneling to connect to a source database, we recommend using a standalone bastion server. This step will walk through provisioning the standalone bastion server. If using an existing server, skip to the next section about configuring network access. 1. In the Azure portal, search for "Virtual machines", and click into the **Virtual Machine** service. Click **Create** and then **Azure virtual machine**. 2. Select an image. We recommend a default Ubuntu image, such as the *Ubuntu Server 20.04 LTS - x64 Gen2* (the default option). 3. Select a type. We recommend a basic & affordable option such as the **Standard\_B1s**. *Because this bastion server will be used exclusively for SSH tunneling, memory, compute, and storage are not particularly important. The Standard\_B1s default configuration comes with sufficient Memory (1 GiB), vCPUs (2) and Storage (8 GB SSD) for about \$7/mo depending on region and market rates.* ### Configure network access 3. Click the **Networking** tab. Select a **Virtual network** to associate with the machine. *By default, you may be prompted to create a new virtual network. You may suggest the default suggestion.* 4. In the **Subnet** field, select the public subnet you want to associate with the virtual machine. *By default, you may be prompted to create a new virtual network. You may accept the default suggestion.* 5. In the **Public IP** dropdown, verify that a valid selection is selected (*do not choose None*). *By default, you may be prompted to create a new Public IP. You may accept the default suggestion.* 6. For NIC network security group, select **Advanced**. This will display the **Configure network security group** dropdown. 7. Click the **Create new** link under the **Configure network security group** dropdown to reveal the Create network security group panel. In the panel, click the **Add an inbound rule** link in the Inbound rules section. This will open the Add inbound security rule panel: 8. Fill in the following fields, leaving any unlisted fields as the default: 1. **Source:** Select the **IP Addresses** option. 2. **Source IP addresses/CIDR ranges:** Add the Prequel static IP. Optionally, add a description. For the correct IP, refer to the [Prequel IPs](/export/deployment/prequel-ips) section of the docs. ***Note**: you will not be able to SSH into this machine to configure the `prequel` user unless you also whitelist your machine's IP in the inbound rules. Consider temporarily adding that IP to facilitate the next step.* 3. **Source port ranges:** Enter **22**. 4. **Protocol:** Select **TCP**. 5. **Action:** Select **Allow**. 6. **Name:** Enter a descriptive name, for example: `prequel-inbound`. 7. When finished, click **Add** to create the inbound rule. 9. Click "OK" and then **Review + create**. Review the details and click **Create** to create the tunnel. ### Make a note of your public and private IP addresses 1. In the Azure portal, search for "Virtual machines", and click into the **Virtual Machine** service. Click the name of the Virtual machine you just created. 2. In the **Networking** section of the **Properties** tab, make a note of the **Public IP address** and **Private IP address**. ### Create the SSH user Log in to the bastion server and complete the steps below. 1. Create group `prequel`: ```shell title="Create group" icon="terminal" theme={null} sudo groupadd prequel ``` 2. Create user `prequel`: ```shell title="Create user" icon="terminal" theme={null} sudo useradd -m -g prequel prequel ``` 3. Switch to the prequel user: ```shell title="Switch user" icon="terminal" theme={null} sudo su - prequel ``` 4. Create the `.ssh` directory: ```shell title="Create .ssh directory" icon="terminal" theme={null} mkdir ~/.ssh ``` 5. Set permissions: ```shell title="Set directory permissions" icon="terminal" theme={null} chmod 700 ~/.ssh ``` 6. Navigate to the `.ssh` directory: ```shell title="Enter .ssh directory" icon="terminal" theme={null} cd ~/.ssh ``` 7. Create the `authorized_keys` file: ```shell title="Create authorized_keys file" icon="terminal" theme={null} touch authorized_keys ``` 8. Set permissions: ```shell title="Set file permissions" icon="terminal" theme={null} chmod 600 authorized_keys ``` 9. Add the public key to the `authorized_keys` file. *The key, including the "ssh-key" and comment, should be all on one line in the file, without linebreaks.* ```shell title="Add public key" icon="terminal" theme={null} echo "ssh-key prequel-ssh-tunnel-public-key" > authorized_keys ``` 1. Find the database you'd like to connect to in your cloud console. 2. Find the **Connection security** section, and add a new rule. 3. If using an **Azure database**, add a new rule with the **Private IP** noted in Step 1 as the **Start IP** and **End IP**. Choose a name and click **Save**. # Create and rotate a service principal PAT (Azure Databricks) Source: https://docs.prequel.co/export/sources/additional-options/databricks-service-principal-pat Create and rotate a Databricks Personal Access Token for a service principal on Azure Databricks. Use this guide to create a Databricks Personal Access Token (PAT) for a service principal on Azure Databricks, and rotate it safely for use with Databricks connections. **Service Principal Types** Azure Databricks supports two types of service principals: * **Azure Databricks managed service principals**: Created and managed directly within Databricks. Can create PATs through the Databricks UI. * **Microsoft Entra ID managed service principals**: Created in Microsoft Entra ID, then imported into Databricks. Accessed via Azure API. This guide covers creating PATs for both **Microsoft Entra ID managed service principals** (indicated with Steps A) and **Databricks managed service principals** (with Steps B). ## Prerequisites * You have Azure admin access to create app registrations in Microsoft Entra ID. * You have Azure Databricks workspace admin access to manage service principals and token permissions. * You know your workspace URL (for example, `https://adb-..azuredatabricks.net`). ## Step 0: choose your service principal type ### Option A: Microsoft Entra ID managed service principal (for cross-Azure authentication) If you need to authenticate with both Azure Databricks and other Azure resources, create a Microsoft Entra ID service principal. If you already have one with the **Application (client) ID**, **Directory (tenant) ID**, and **client secret**, you can [skip to Step 2A](#step-2a-add-entra-id-service-principal-to-databricks). ## Step 1A: create the app registration in Microsoft Entra ID 1. Sign in to the **Azure portal**. 2. Navigate to **Microsoft Entra ID** → **App registrations** → **New registration**. ![](https://storage.googleapis.com/prequel_docs/images/azure-databricks-create-app-registration.png "azure_databricks_create_app_registration.png") 3. Enter a **Name** (e.g., "Databricks Service Principal"). 4. Under **Supported account types**, select **Accounts in this organizational directory only (Single tenant)**. 5. Click **Register**. 6. On the **Overview** page, copy and save: * **Application (client) ID** * **Directory (tenant) ID** #### Create a client secret 1. In the app registration, go to **Certificates & secrets** → **Client secrets** → **New client secret**. 2. Enter a **Description** and select an **Expires** duration (e.g., 12 months). 3. Click **Add**. 4. **Important**: Copy and securely store the **Value**. This is your client secret and will not be shown again. ## Step 2A: add Entra ID service principal to Databricks ### Import the service principal 1. In your Databricks workspace, click your username → **Settings**. 2. Go to **Identity and access** → **Service principals** → **Add service principal**. 3. Select **Import service principal**. 4. Paste the **Application (client) ID** from Step 1. 5. Click **Add**. ### Grant workspace entitlements 1. Click on the newly created service principal. 2. Under **Entitlements**, enable: * **Workspace access** * **Databricks SQL access** (if needed for SQL Warehouse access) 3. Click **Update**. Now continue to [Step 3A](#step-3a-configure-token-permissions-for-entra-id-sp). ## Step 3A: configure token permissions for Entra ID SP ### Enable PAT usage for the service principal 1. Configure token permissions via API using an existing admin PAT: ```bash title="Configure token permissions" icon="terminal" expandable theme={null} curl --request PATCH 'https://adb-..azuredatabricks.net/api/2.0/preview/permissions/authorization/tokens' \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data-raw '{ "access_control_list": [ { "service_principal_name": "", "permission_level": "CAN_USE" } ] }' ``` Replace `` with your service principal's Application (client) ID from Step 1. **Note: PAT workspace setting** If PAT authentication is disabled at the workspace level, users and service principals cannot create or use PATs until re-enabled. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/access-control/tokens)) Continue to [Step 4A](#step-4a-generate-microsoft-entra-id-access-token). ## Step 4A: generate Microsoft Entra ID access token Use the service principal credentials to request an Entra ID token: ```bash title="Generate Entra ID token" icon="terminal" expandable theme={null} # Replace values in <>. Token expires in ~1 hour. curl -X POST "https://login.microsoftonline.com//oauth2/v2.0/token" \ -H 'Content-Type: application/x-www-form-urlencoded' \ -d 'client_id=' \ -d 'client_secret=' \ -d 'grant_type=client_credentials' \ -d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default' ``` Replace: * `` with the **Directory (tenant) ID** from Step 1 * `` with the **Application (client) ID** from Step 1 * `` with the client secret **Value** from Step 1 The `2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default` scope identifies Azure Databricks and is not workspace-specific. Do not change it. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/service-prin-aad-token)) **Token lifetime** Entra tokens are short-lived (\~1 hour). Use immediately to create the PAT, or automate with CLI/SDK tools. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/service-prin-aad-token)) Continue to [Step 5A](#step-5a-create-pat-for-entra-id-sp-via-api). ## Step 5A: create PAT for Entra ID SP via API **Microsoft Entra ID service principals can only create PATs through API calls, not through the Databricks UI.** This is because they are API-only identities that cannot log into the workspace interface. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/pat)) Call the Databricks Tokens API using the workspace URL and Entra access token from Step 4: ```bash title="Create PAT" icon="terminal" expandable theme={null} # Create a PAT for the service principal identity curl -X POST "https://adb-..azuredatabricks.net/api/2.0/token/create" \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{ "lifetime_seconds": 31536000, "comment": "Databricks data pipeline (service principal)" }' ``` The response includes a `token_value` that starts with `dapi...`. Copy and store it securely; you will not see it again. ([Databricks Documentation](https://docs.databricks.com/en/dev-tools/api/latest/tokens.html)) **Alternative: Databricks CLI approach** You can also create the PAT with the Databricks CLI (`databricks tokens create`) after authenticating the service principal via OAuth M2M or Entra SP auth. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication)) ### Option B: Databricks managed service principal (Databricks-only) If you only need Databricks access, you can create a Databricks managed service principal directly in the UI. [Skip to Step 2B](#step-2b-create-databricks-managed-service-principal) for this flow. ## Step 2B: create Databricks managed service principal 1. In your Databricks workspace, click your username → **Settings**. 2. Go to **Identity and access** → **Service principals** → **Manage**. 3. Click **Add service principal** → **Add new**. 4. Enter a **Display name** and click **Add**. 5. Click on the newly created service principal and under **Entitlements**, enable: * **Workspace access** * **Databricks SQL access** (if needed for SQL Warehouse access) 6. Click **Update** and make a note of the **Application ID**. Continue to [Step 3B](#step-3b-configure-token-permissions-for-databricks-sp). ## Step 3B: configure token permissions for Databricks SP 1. In **Admin Settings** → **Access control** → **Personal access tokens** → **Permission settings**. 2. Search for and select your Databricks service principal. 3. Grant **CAN USE** permission. 4. Click **Add** and **Save**. Continue to [Step 4B](#step-4b-create-pat-for-databricks-sp-via-api). ## Step 4B: create PAT for Databricks SP via API **For Databricks managed service principals, you can create a PAT using the on-behalf-of tokens API with an admin user PAT.** You'll need an existing admin Personal Access Token to create a token on behalf of the service principal: ```bash title="Create PAT on behalf of SP" icon="terminal" expandable theme={null} curl --request POST "https:///api/2.0/token-management/on-behalf-of/tokens" \ --header "Authorization: Bearer " \ --header "Content-Type: application/json" \ --data '{ "application_id": "", "lifetime_seconds": 31536000, "comment": "Databricks data pipeline (service principal)" }' ``` Replace: * `` with your workspace URL * `` with a workspace admin's PAT * `` with the Application ID from Step 2B **Alternative: UI approach** Some Databricks managed service principals may also create PATs through **Admin Settings** → **Identity and access** → **Service principals** → click service principal → **Access tokens** → **Generate new token**, depending on workspace configuration. Continue to [Step 6](#step-6-use-the-pat-in-your-integration). ## Step 6: use the PAT in your integration Use this token value in your Databricks connection configuration (Personal access token). ## Rotation procedure 1. Create a new PAT for the service principal (repeat Steps 4-5). 2. Update your integration's Databricks configuration with the new PAT. 3. Revoke the old PAT using the UI or API: ```bash title="Revoke PAT" icon="terminal" theme={null} # Using CLI: delete by token ID databricks tokens delete ``` (Or use the Token Management API to delete a token by ID.) ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/access-control/tokens), [Databricks Documentation](https://docs.databricks.com/en/dev-tools/api/latest/token-management.html)) ## Troubleshooting ### 403 / not authorized when creating PAT PATs may be disabled for the workspace, or the service principal/group lacks **CAN USE** permission. Check **Admin Settings** → **Access control** → **Personal access tokens**. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/access-control/tokens)) ### Invalid scope when requesting Entra token Use the exact scope `2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default`. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/service-prin-aad-token)) ### Account-level APIs failing with PAT PATs are for workspace-level auth. Account-level automation requires Entra/OAuth tokens. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/access-control/tokens)) # GCP SSH tunneling Source: https://docs.prequel.co/export/sources/additional-options/gcp-ssh-tunneling Configure a GCP bastion server to accept an SSH connection from Prequel. **Coming soon!** Contact us for this documentation section. # Google Cloud Storage staging bucket Source: https://docs.prequel.co/export/sources/additional-options/google-cloud-storage-staging-bucket Set up a GCS staging bucket for efficient source or destination data transfer. Some sources or destinations without built-in staging resources require a staging bucket to efficiently transfer or ingest data. ### Create a service account 1. In the GCP console, navigate to the **IAM & Admin** menu. 2. Click into the **Service Accounts** tab. 3. Click **Create service account** at the top of the menu. 4. In the first step, name the service account that will be used to transfer data into Cloud Storage and click **Create and Continue**. You may continue through the following steps without assigning any roles, and in the final step, click **Done**. ### Create staging bucket 1. Navigate to the **Cloud Storage** page. 2. Click **Create**. 3. Enter a **bucket name**, choose a **region**. 4. After choosing your preferences for the remaining steps, click **Create**. *(If presented with a warning, you may enforce public access prevention)* 5. On the **Bucket details** page for the bucket you created, select the **Permissions** tab, and click **Grant access**. 6. Grant access to the principal (Service Account) you created in **Step 1**, and assign the Roles: **Storage Object Creator** and **Storage Object Viewer**. Click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcs-storage-object-creator-viewer.png) **Optional: Add a short retention lifecycle policy** You may configure a lifecycle rule on the staging bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. In the bucket **Lifecycle** tab, add a rule with action "Delete object" and condition "Age: 2 days". Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ### Generate HMAC key An HMAC key is a type of credential and can be associated with a service account or a user account to access Google Cloud Storage. 1. Navigate to the **Cloud Storage** page. 2. Click into the **Settings** tab on the left side menu. 3. In the **Interoperability** tab, click the **Create a key for a Service Account** button. 4. Select the **Service Account** created in **Step 1**, and click **Create key**. ![](https://storage.googleapis.com/prequel_docs/images/gcs-select-service-account-hmac.png) 5. Make a note of the **Access key** and **Secret**. ### You're done! Use this configured GCS staging bucket during the connection of your preferred data source or destination. **Note**: depending on the connection method, you may need to refer to the following field name mappings: * **Bucket Access ID**: your HMAC Access Key. * **Bucket Secret Key**: your HMAC Secret. # S3 staging bucket Source: https://docs.prequel.co/export/sources/additional-options/s3-staging-bucket Set up an S3 staging bucket for efficient source or destination data transfer. Some sources or destinations without built-in staging resources require a staging bucket to efficiently transfer or ingest data. ### Prerequisites * By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` ### Create staging bucket 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "**ACLs disabled**" and **Block Public Access settings for this bucket** can be set to "**Block all public access**" as recommended by AWS. Make note of the Bucket name and AWS Region. 4. Click **Create bucket**. **Optional: Add a short retention lifecycle policy** You may configure a lifecycle rule on the staging bucket to automatically delete objects older than 2 days as the bucket is not used to persist data. In the bucket **Management** tab, click **Create lifecycle rule**, set an expiration action for current versions of objects with a 2-day age. Note that transfer logic automatically cleans up files after transfer completion, so this is an optional step. ### Create policy 5. Navigate to the **IAM** service page, click on the **Policies** navigation tab, and click **Create policy**. 6. Click the JSON tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen above. 1. **Note**: the first policy applies to `BUCKET_NAME` whereas the second policy applies only to the bucket's contents, `BUCKET_NAME/*`, an important distinction. ```json title="Bucket access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` **KMS encryption (optional)** If your S3 staging bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 7. Click through to the **Review** step, choose a **name** for the policy, for example, `transfer-service-policy` (this will be referenced in the next step), add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. ### You're done! Use this configured S3 staging bucket during the connection of your preferred data source or destination. # AWS SSH tunneling Source: https://docs.prequel.co/export/sources/additional-options/ssh-tunneling Configure an AWS bastion server to accept an SSH connection from Prequel. Follow these steps to configure an AWS bastion server to accept an SSH connection from Prequel. ### Prerequisites * A public SSH key to add to the bastion machine to authenticate the SSH connection. *Ask your contact at Prequel if they have not already provided one to you.* If using SSH tunneling to connect to a source database, we recommend using a standalone bastion server. This step will walk through provisioning the standalone bastion server. If using an existing server, skip to step 2. 1. Navigate to the EC2 Instances page, and click **Launch instances**. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-launch-instances.png "launch instances.png") 2. Select an image. We recommend a default Amazon Linux image, such as the *64-bit (x86) Amazon Linux 2 AMI* (the default option). ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-select-image.png "image.png") 3. Select a type. We recommend the **t3.nano**. *Because this bastion server will be used exclusively for SSH tunneling, Network Performance is the primary consideration. The t3.nano default configuration comes with Network Performance of up to 5 Gigabit, as well as minimal Memory (1 GiB), vCPUs (2) and Storage (8 GB SSD) for about \$5/mo depending on region and market rates.* ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-t3-nano.png "t3nano.png") ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-t3-nano-cost.png "t3nano cost.png") 4. Click **Review and launch**. Consider modifying the default security group according to your security policies to enable configuration (e.g., creating the `prequel` user in **Step 2**). Click **Launch**. We will further modify the default security group in **Step 2**. (The remaining default options are sufficient.) ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-select-launch.png "launch.png") ### Allow SSH port access In the AWS console, complete the steps below 1. Navigate to the **EC2 Instances** page, and click the **instance ID** of the server to be used as the bastion host. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-instance-id.png "ec2 instance id 2.png") 2. In the **Security** tab, click the link to one of the security groups. (E.g., the **default** security group). Make note of this security group, as you will need to allow inbound access to your source DB as part of **Step 3**. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-security-tab.png "instance security tab.png") 3. In the **Inbound rules** tab, click "Edit inbound rules". ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-inbound-rules.png "ec2 edit inbound rules.png") 4. Click **Add rule**. Add an inbound rule of type **SSH** and whitelist the Prequel static IP. Optionally, add a description. For the correct IP, refer to the [Prequel IPs](/export/deployment/prequel-ips) section of the docs. ***Note**: you will not be able to SSH into this machine to configure the `prequel` user unless you also whitelist your machine's IP in the inbound rules. Consider temporarily adding that IP to facilitate the next step.* ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-whitelist-prequel-ip.png "ssh tunnel whitelist.png") ### Create the SSH user Log in to the bastion server and complete the steps below. 1. Create group `prequel`: ```shell title="Create group" icon="terminal" theme={null} sudo groupadd prequel ``` 2. Create user `prequel`: ```shell title="Create user" icon="terminal" theme={null} sudo useradd -m -g prequel prequel ``` 3. Switch to the prequel user: ```shell title="Switch user" icon="terminal" theme={null} sudo su - prequel ``` 4. Create the `.ssh` directory: ```shell title="Create .ssh directory" icon="terminal" theme={null} mkdir ~/.ssh ``` 5. Set permissions: ```shell title="Set directory permissions" icon="terminal" theme={null} chmod 700 ~/.ssh ``` 6. Navigate to the `.ssh` directory: ```shell title="Enter .ssh directory" icon="terminal" theme={null} cd ~/.ssh ``` 7. Create the `authorized_keys` file: ```shell title="Create authorized_keys file" icon="terminal" theme={null} touch authorized_keys ``` 8. Set permissions: ```shell title="Set file permissions" icon="terminal" theme={null} chmod 600 authorized_keys ``` 9. Add the public key to the `authorized_keys` file. *The key, including the "ssh-key" and comment, should be all on one line in the file, without linebreaks.* ```shell title="Add public key" icon="terminal" theme={null} echo "ssh-key prequel-ssh-tunnel-public-key" > authorized_keys ``` 1. Within the **Inbound rules** of the database security group, click **Add rule**. Add the security group noted in **Step 2**, and click **Save rules**. ![](https://storage.googleapis.com/prequel_docs/images/ssh-tunneling-bastion-rule.png "Screen Shot 2022-01-13 at 2.53.12 PM.png") # Athena Source: https://docs.prequel.co/export/sources/athena Instructions for connecting to an AWS Athena service and the underlying S3 storage as a source ## Prerequisites * By default, Athena authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier: ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` **Trust policies for AWS deployments** If you are running Prequel on a self-hosted AWS deployment, the Trust Policy JSON will reference an AWS federation account instead of the Google Cloud federation service seen in the above JSON document. ### Create Athena staging bucket Follow these steps to create a bucket to be used for staging data before transferring to a destination. 1. Navigate to the **S3** service page. 2. Click **Create bucket**. 3. Enter a **Bucket name**, select the **same AWS Region** as your Athena S3 source, and modify any of the default settings as desired. Note: **Object Ownership** can be set to "**ACLs disabled**" and **Block Public Access settings for this bucket** can be set to "**Block all public access**" as recommended by AWS. Make note of the Bucket name and AWS Region. 4. Click **Create bucket**. ### Create Athena access policy 1. Navigate to the **IAM** service page, click on the **Policies** navigation tab, and click **Create policy**. 2. Click the JSON tab, and paste the following policy, being sure to replace `ACCOUNT_ID`, `WORKGROUP`, `BUCKET_SOURCE` and `BUCKET_STAGING` with the your account information. 1. `WORKGROUP` should be `primary` unless otherwise specified during connection configuration. 2. `BUCKET_SOURCE` should refer to the bucket(s) containing the underlying data to be queried by Athena. 3. `BUCKET_STAGING` should refer to the staging bucket created in the previous step. Note: a `staging` folder is created automatically in the provided staging bucket to temporarily "stage" query results before transferring to destinations. ```json title="Access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowAthenaAccess", "Effect": "Allow", "Action": [ "athena:GetQueryResults", "athena:StartQueryExecution", "athena:StopQueryExecution", "athena:StartSession", "athena:GetDatabase", "athena:GetDataCatalog", "athena:GetWorkGroup", "athena:GetTableMetadata", "athena:GetQueryExecution" ], "Resource": [ "arn:aws:athena:*:ACCOUNT_ID:workgroup/WORKGROUP" ] }, { "Sid": "AllowGlueAccessToSourceDatabaseAndTables", "Effect": "Allow", "Action": [ "glue:GetDatabases", "glue:GetDatabase", "glue:GetTables", "glue:GetTable", "glue:GetPartitions", "glue:CreateTable", "glue:CreateDatabase", "glue:UpdateTable" ], "Resource": [ "arn:aws:glue:*:ACCOUNT_ID:catalog", "arn:aws:glue:*:ACCOUNT_ID:database/DATABASE_SOURCE", "arn:aws:glue:*:ACCOUNT_ID:database/default", "arn:aws:glue:*:ACCOUNT_ID:table/DATABASE_SOURCE/*" ] }, { "Sid": "AllowGlueAccessToDeleteSnapshotTables", "Effect": "Allow", "Action": [ "glue:DeleteTable" ], "Resource": [ "arn:aws:glue:*:ACCOUNT_ID:catalog", "arn:aws:glue:*:ACCOUNT_ID:database/DATABASE_SOURCE", "arn:aws:glue:*:ACCOUNT_ID:table/DATABASE_SOURCE/prequel*" ] }, { "Sid": "AllowS3AccessToSourceBucket", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::BUCKET_SOURCE", "arn:aws:s3:::BUCKET_SOURCE/*" ] }, { "Sid": "AllowS3AccessToStagingBucket", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::BUCKET_STAGING", "arn:aws:s3:::BUCKET_STAGING/*" ] } ] } ``` **KMS encryption (optional)** If your S3 source or staging buckets use KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. **Athena vs. S3 permissions** Because Athena uses S3 as a data source, the Resource access requested in the policy are scoped down via resource-specific permissions in the S3 actions. **`BUCKET_SOURCE` vs. `BUCKET_STAGING`** During source configuration, you will notice both a `BUCKET_SOURCE` and a `BUCKET_STAGING`. They are used for two different purposes here: * `BUCKET_SOURCE` should be the name of the S3 bucket in which the source data is located. Permissions on this resource are limited to read-only operations. * `BUCKET_STAGING` should be the name of the S3 bucket in which the Athena query results are written (i.e., the automatically generated `athena_output/` data). 3. Click through to the **Review** step, choose a **name** for the policy, for example, `transfer-service-policy` (this will be referenced in the next step), add a description, and click **Create policy**. ### Create role 1. Navigate to the **IAM** service page. 2. Navigate to the **Roles** navigation tab, and click **Create role**. 3. Select **Custom trust policy** and paste the provided trust policy (referenced in the prerequisite at the top of this guide) to allow **AssumeRole** access to this role. Click **Next**. 4. Add the permissions policy created above, and click **Next**. 5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**. 6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value. Role based authentication is the preferred authentication mode for Athena based on AWS recommendations. However, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred. The staging bucket and access policy steps from the recommended tab still apply. To authenticate with an AWS User instead of a role: 1. Navigate to the **IAM** service page. 2. Navigate to the **Users** navigation tab, and click **Add users**. 3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**. 4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**. 5. Click **Next: Review** and click **Create user**. 6. In the **Success** screen, record the **Access key ID** and the **Secret access key**. Test the connection & add the configured source to Prequel in the Prequel UI or via [cURL request](/export/api-reference/overview). # BigQuery Source: https://docs.prequel.co/export/sources/bigquery Steps to add a BigQuery data warehouse as a data source. 1. In the GCP console, in the same project as your BigQuery instance, navigate to the **IAM & Admin** menu. ![](https://storage.googleapis.com/prequel_docs/images/gcp-iam-admin-full.png "iam & admin full.png") 2. Click into the **Service Accounts** tab. ![](https://storage.googleapis.com/prequel_docs/images/gcp-iam-admin-full.png "service accounts full.png") 3. Click **Create service account** at the top of the menu. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-service-account-menu.png "create service account menu.png") 4. In the first step, name the user and click **Create and Continue**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-service-account-name-options.png "service account name.png") 5. In the second step, grant the user the **BigQuery User** role. ![](https://storage.googleapis.com/prequel_docs/images/gcp-bigquery-user.png) 6. In the third and final step, click **Done**. 7. Back in the **Service accounts** menu, click the Actions dropdown next to the newly created service account and click **Manage keys**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-manage-service-account-keys.png "manage keys.png") 8. Click **Add key** and then **Create new key**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-new-key.png "create new key.png") 9. Select the **JSON** Key type and click **Create** and make note of the key that is generated. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-json-key.png "json create.png") Transferring data from BigQuery requires a temporary staging area within Google Cloud Storage to stage compressed data before copying to a destination. 1. Log into the Google Cloud Console and navigate to **Cloud Storage**. Click **Create** to create a new bucket. ![](https://storage.googleapis.com/prequel_docs/images/gcp-create-gcs-bucket.png) 2. Choose a **name** for the bucket. Click **Continue**. Select a **location** for the staging bucket. Make a note of both the **name** and the **location** (region). **Choosing a `location` (region)** The location you choose for your staging bucket must match the location of your source dataset in BigQuery. 3. Click **continue** and complete the options that follow according to your preferences. Once the options have been filled out, click **Create**. 4. On the **Bucket details** page that appears, click the **Permissions** tab, and then click **Add**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-add-permission-to-bucket.png) 5. In the **New principles** dropdown, add the Service Account created in **Step 1**, select the **Storage Admin** role, and click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/gcp-storage-admin.png) 1. Navigate to your **BigQuery** project and click on the dataset you want to share. Make a note of the **Project ID** and **Data location**. 2. In the menu on the right, click **Sharing** and then click **Permissions**. ![](https://storage.googleapis.com/prequel_docs/images/bigquery-sharing-permissions.png) 3. Click **Add Principal**. ![](https://storage.googleapis.com/prequel_docs/images/bigquery-add-principal.png) 4. Select the Service Account created in **Step 1**, then select the **BigQuery User** role and the **BigQuery Data Viewer** role. Click **Save**. ![](https://storage.googleapis.com/prequel_docs/images/bigquery-user-data-viewer.png) 1. Use the **Project ID**, **Data Location**, **Source Dataset Name**, **Bucket Name**, **Bucket Region**, and **Service Account Private Key** (`.json` file) to [add the Source](/export/api-reference/overview) using the Prequel API. # ClickHouse Source: https://docs.prequel.co/export/sources/clickhouse Connect a ClickHouse data warehouse as a Prequel source. For setup instructions, see [Generic ClickHouse](/export/sources/clickhouse-generic). # Generic ClickHouse Source: https://docs.prequel.co/export/sources/clickhouse-generic Instructions for connecting to a ClickHouse data warehouse as a source 1. Make a note of your [Prequel static IP](/export/deployment/prequel-ips) 2. Create a rule in a security group or firewall settings to whitelist: 1. incoming connections to your host and port (usually `9440`) from the static IP. 2. outgoing connections from ports `1024` to `65535` to the static IP. Create a database user to perform the reading of the source data. 1. Open a connection to your ClickHouse database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create reader user" icon="database" theme={null} CREATE USER @'%' IDENTIFIED BY ''; ``` 3. Grant user required privileges on the database. **Additional permissions required for source queries** If any of your Prequel models currently or will use source queries, you will need to also provide `CREATE VIEW` and `DROP VIEW` privileges to the Clickhouse user below. ```sql title="Grant privileges" icon="database" theme={null} GRANT SELECT ON <{database.table|database.*|*.*}> TO @'%'; GRANT CREATE TEMPORARY TABLE, S3 on *.* TO @'%'; ``` **Understanding the `CREATE TEMPORARY TABLE, S3` permissions** The `CREATE TEMPORARY TABLE` and `S3` permissions are required to efficiently transfer data from ClickHouse. Under the hood, these permissions are used to stage data in a temporary table and export compressed data into object storage for transferring. By definition, the temporary table will not exist outside of the session. ClickHouse sources require a staging bucket to efficiently transfer data. Configure your staging bucket using one of the following guides: * [S3 staging bucket configuration](/export/sources/additional-options/s3-staging-bucket) * [Google Cloud Storage staging bucket configuration](/export/sources/additional-options/google-cloud-storage-staging-bucket) ### Optional: granting ClickHouse Cloud role-based access to S3 If your ClickHouse instance runs on ClickHouse Cloud, you can have it authenticate to your S3 staging bucket using the same IAM role instead of access keys to avoid relying on long-lived static credentials. You may follow the same steps in the [S3 staging bucket configuration](/export/sources/additional-options/s3-staging-bucket) above, but you will need to add an additional trust policy statement to allow ClickHouse to assume the role too. ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { // Existing statements }, { "Effect": "Allow", "Principal": {"AWS": ""}, "Action": "sts:AssumeRole" } ] } ``` Replace `` with your ClickHouse instance's IAM ARN. To obtain the ARN, go to your ClickHouse Cloud account, navigate to **Settings** → **Network security information** → **View service details** and copy the **Service role ID (IAM)**. See the [ClickHouse Secure S3 documentation](https://clickhouse.com/docs/cloud/data-sources/secure-s3) for full details, including an automated CloudFormation setup option. Use the [cURL request](/export/api-reference/sources/create-source) to add the configured ClickHouse source and staging bucket. # Databricks Source: https://docs.prequel.co/export/sources/databricks Instructions for connecting your Databricks workspace as a source. ## Prerequisites * An existing SQL Warehouse in your Databricks workspace that you want to connect to Prequel. You'll need to access its **Connection details** to get the **Server hostname**, **Port**, and **HTTP Path**. ([Databricks Documentation](https://docs.databricks.com/en/compute/sql-warehouse/create.html), [Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/sql/admin/sql-endpoints)) * Permissions to create a Personal Access Token (PAT) in your Databricks workspace. You can use a user PAT or a service-principal PAT. **Step #2** below describes how to generate a user PAT. For a service-principal PAT, see our guide on [Create and rotate a service principal PAT (Azure Databricks)](/export/sources/additional-options/databricks-service-principal-pat) * **For Hive Metastore users**: An S3 staging bucket with proper IAM credentials that your SQL warehouse can write to and that Prequel can access for temporary data staging during transfer. See **Step #3B** below for detailed setup instructions. * **Optional networking**: If your organization restricts inbound traffic, ensure HTTPS 443 access to your workspace hostname is allowed for Prequel's connector. ([Databricks Documentation](https://docs.databricks.com/en/security/network/ip-access-list.html)) 1. In your Databricks workspace, go to **SQL** → **SQL Warehouses** and select the warehouse you want to connect to Prequel. 2. Click **Connection details** and copy the **Server hostname**, **Port** (typically 443), and **HTTP Path**. ([Databricks Documentation](https://docs.databricks.com/en/compute/sql-warehouse/create.html), [Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/sql/admin/sql-endpoints)). Note that the port is contained within the **JDBC url** populated. ![](https://storage.googleapis.com/prequel_docs/images/databricks-create-a-databricks-pat-2025.png "create_a_personal_access_token.png") 1. Create or use an existing Databricks service principal in your workspace and generate a PAT for the service principal. See our comprehensive [Create and rotate a service principal PAT](/export/sources/additional-options/databricks-service-principal-pat) guide. ([Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/administration-guide/users-groups/service-principals)) 1. Then, on the same screen, select **Create a Personal Access Token**. ([Databricks Documentation](https://docs.databricks.com/en/dev-tools/auth/pat.html)) Your Databricks workspace uses one of two metastore types. Follow the corresponding steps based on your metastore: ### A. Unity Catalog (recommended) If your data is in Unity Catalog: * No external staging bucket required. Prequel will create temporary Unity Catalog volumes for data staging and clean them up automatically. ([Databricks Documentation](https://docs.databricks.com/en/data-governance/unity-catalog/index.html), [Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/)) * You'll need to specify which catalog and optionally which schema contains your data. ### B. Hive Metastore If your data is in the Hive Metastore: * You'll need to provide an S3 staging bucket that your SQL warehouse can write to and that Prequel can access for temporary data staging during transfer. * You'll also need AWS access credentials (Access Key ID and Secret Access Key) that have permissions to read/write to this staging bucket. #### Setup S3 staging bucket for Hive Metastore Follow these steps to create and configure the required S3 staging bucket: 1. **Create the staging bucket**: * Navigate to the **S3** service page in AWS console * Click **Create bucket** * Enter a **Bucket name** and select an **AWS Region** (select the same region as your Databricks workspace) * Set **Object Ownership** to "**ACLs disabled**" and **Block Public Access settings** to "**Block all public access**" as recommended by AWS. Leave the default encryption method of **SSE-S3** selected. * Click **Create bucket** 2. **Create IAM policy for bucket access**: * Navigate to the **IAM** service page, click **Policies**, and click **Create policy** * Click the **JSON** tab and paste the following policy, replacing `BUCKET_NAME` with your staging bucket name: ```json title="Bucket access policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` **KMS encryption (optional)** If your S3 staging bucket uses KMS encryption (CMK), add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key. Encryption with SSE-C is not currently supported. ```json title="KMS statement" icon="brackets-curly" expandable theme={null} { "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. 3. **Create IAM user and access keys**: * Navigate to **IAM** → **Users** and click **Create user** * Enter a **User name** (e.g., `databricks-prequel-staging`) * Click **Next**, attach the policy created above, and click **Create user** * Click on the created user, go to **Security credentials** tab * Click **Create access key**, choose **Application running outside AWS**, and create the key * **Important**: Save the **Access Key ID** and **Secret Access Key** - you'll need these for Prequel configuration In Prequel, create a new Databricks source with the following configuration: * **Name**: Descriptive name for your Databricks source * **Vendor**: `databricks` * **Server hostname**: From Step 1 * **Port**: `443` (from Step 1) * **HTTP Path**: From Step 1 * **Authentication**: Personal Access Token (from Step 2) * **Metastore**: Select `unity_catalog` or `hive` based on your setup ### For Unity Catalog workspaces * **Catalog**: Name of the catalog containing your data * **Schema** (optional): Default schema to use ### For Hive Metastore workspaces * **Database**: Name of the database containing your data * **S3 staging bucket**: * **Bucket name**: Name of the S3 staging bucket created above * **Bucket region**: AWS region of the staging bucket * **Access Key ID**: AWS access key ID with permissions to the staging bucket * **Secret Access Key**: AWS secret access key corresponding to the access key ID 1. In Prequel, click **Test connection** to verify the setup. 2. If your SQL warehouse is currently stopped, it will restart automatically when Prequel runs a test query. Ensure your PAT has permission to use the specified warehouse. ([Databricks Documentation](https://docs.databricks.com/en/compute/sql-warehouse/create.html), [Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/sql/admin/sql-endpoints)) ## Notes and known behaviors * If your tables are in the Hive Metastore, make sure to select "hive" as the metastore type, as `information_schema` does not list `hive_metastore` objects. ([Databricks Documentation](https://docs.databricks.com/en/sql/language-manual/sql-ref-information-schema.html)) * Unity Catalog volumes used for staging are temporary and automatically cleaned up after each transfer. ([Databricks Documentation](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-volume.html)) ## Related guides Generate and rotate a personal access token for an Azure Databricks service principal. # MongoDB Source: https://docs.prequel.co/export/sources/mongodb Instructions for connecting to a MongoDB replica set to share data. ### MongoDB CLI 1. Connect to your **MongoDB node** using the **MongoDB CLI** as an **admin**. 2. Execute the following query: ```text title="MongoDB Script" theme={null} db.adminCommand( { replSetGetStatus : 1 } ).members ``` 3. Make a note of the **host identifiers**. ### MongoDB Atlas 1. Log in to your Atlas **dashboard**, click into the **Database** tab and click **Connect**. 2. Under **Connect your application**, click into the **Drivers** option. 3. In the following screen, make a note of the **host identifier** within your connection string. It can be found within the string that starts with `mongodb+srv` and the identifier is the URI after the `@`. E.g., `some-cluster.some-characters.mongodb.net`. ### MondoDB CLI 1. Connect to your MongoDB node using the MongoDB CLI as an admin. 2. Execute the following script (with a `username` and `password`) to create a new user. ```text title="MongoDB Script" theme={null} use admin db.createUser({ user: "", pwd: "", roles: [ {role: "read", db: ""} ] }) ``` ### MongoDB Atlas 1. Log in to your Atlas **dashboard**, click into the **Database Access** section of the **Security** options. 2. Select **Add New Database User**. Choose the **Password** authentication method, and enter a `username` and `password` for the new user. 3. In the **Database User Privileges** menu, select **Grant Specific User Privileges**. Within **Specific Privileges**, add the following: `read@` (for each database you'd like to grant access to). Leave the **Collection** field blank. 4. Click **Add User** to create the new user. If your MongoDB instance enforces IP Access restrictions, you'll need to allow access to a static IP. ### MongoDB CLI 1. Connect to your MongoDB node using the MongoDB CLI as an admin. 2. Execute the following script ```text title="MongoDB Script" theme={null} use admin atlas accessLists create "" --comment "Prequel data sharing platform static IP" ``` ### MongoDB Atlas 1. Log in to your Atlas **dashboard**, click into the **Network Access** section of the **Security** options. 2. Select **Add IP Address**. In **Access List Entry** enter your Prequel deployment's static IP. [See here](/export/deployment/prequel-ips) for the list of static IPs by deployment. 3. Add an optional comment (e.g., "Prequel data sharing platform") and click **Confirm**. Use the **host name**, **port**, and **database name** (as noted in **Step 1**) and the **username** and **password** (as configured in **Step 2**) to complete the connection. # MySQL Source: https://docs.prequel.co/export/sources/mysql Configuring your MySQL source. ## Prerequisites * If your MySQL database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Create a rule in a security group or firewall settings to whitelist: * incoming connections to your host and port (usually `3306`) from the static IP. * outgoing connections from ports `1024` to `65535` to the static IP. Create a database user to perform the reading of the source data. 1. Open a connection to your MySQL database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create reader user" icon="database" theme={null} CREATE USER @'%' IDENTIFIED BY ''; ``` 3. Grant user required privileges on the relevant tables. ```sql title="Grant table privileges" icon="database" theme={null} GRANT SELECT ON 'some_schema'.'some_table' TO @'%'; ``` Securely share your **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password** with us to complete the connection. # Oracle Source: https://docs.prequel.co/export/sources/oracle Configuring your Oracle source. ## Prerequisites * If your Oracle database is protected by security groups or other firewall settings, you will need to have our static IP available to complete Step 1. Create a rule in a security group or firewall settings to whitelist: * incoming connections to your host and port (usually `1521`) from the static IP. * outgoing connections from ports `1024` to `65535` to the static IP. Create a database user to perform the reading of the source data. 1. Open a connection to your Oracle database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create reader user" icon="database" theme={null} CREATE USER IDENTIFIED BY ''; GRANT CREATE SESSION TO ; ``` 3. Grant user required privileges on the relevant tables. ```sql title="Grant table privileges" icon="database" theme={null} GRANT SELECT ON 'some_schema'.'some_table' TO ; ``` Securely share your **host name**, **database name**, **port**, your chosen **schema name**, **username**, and **password** with us to complete the connection. # Overview Source: https://docs.prequel.co/export/sources/overview Documentation and instructions for configuring sources of data Select your source vendor below for connection setup instructions. For an overview of how sources work in Prequel, see [Sources](/export/concepts/sources). ## Supported sources ### Data warehouses | Vendor | Status | Docs | | :------------------- | :----- | :----------------------------------------- | | Snowflake | GA | [link](/export/sources/snowflake) | | BigQuery | GA | [link](/export/sources/bigquery) | | Redshift | GA | [link](/export/sources/redshift) | | Databricks | GA | [link](/export/sources/databricks) | | Athena | GA | [link](/export/sources/athena) | | ClickHouse | GA | [link](/export/sources/clickhouse) | | ClickHouse (generic) | GA | [link](/export/sources/clickhouse-generic) | ### Databases | Vendor | Status | Docs | | :-------------- | :----- | :------------------------------------------ | | Postgres | GA | [link](/export/sources/postgres) | | Aurora Postgres | GA | [link](/export/sources/postgres-aws-aurora) | | MySQL | GA | [link](/export/sources/mysql) | | MongoDB | GA | [link](/export/sources/mongodb) | | Oracle | GA | [link](/export/sources/oracle) | | SQL Server | Beta | [link](/export/sources/sql-server) | # Postgres Source: https://docs.prequel.co/export/sources/postgres Instructions for connecting to a Postgres database as a source We support all flavors of Postgres. That said, the configuration on the database depends on the cloud provider it's running on. See below for detailed instructions for: * [AWS Aurora](/export/sources/postgres-aws-aurora) * [AWS RDS](/export/sources/postgres-aws-rds) * [GCP Postgres](/export/sources/postgres-gcp) # AWS Aurora Source: https://docs.prequel.co/export/sources/postgres-aws-aurora Configure an AWS Aurora PostgreSQL database as a Prequel source. Follow these steps to configure your source database and connect to Prequel. **Step 1** (creating a read-only endpoint) is optional, but will ensure the Prequel service does not put any unnecessary load on your primary database during data transfer. 1. In your Amazon RDS Dashboard, click the Aurora PostgreSQL instance to which you want to add a reader. 2. On the database page, click **Actions**, then select **Add reader** from the drop down. ![](https://storage.googleapis.com/prequel_docs/images/aurora-add-reader.png "add reader.png") 3. In the **Settings** section, enter a **DB instance identifier**. For example, `source-transfer-service-reader`. 4. In the **DB instance class** sections, specify the instance type for the read replica. It can be smaller than the main instance, though AWS may not allow anything smaller. 5. If you are connecting directly: in the **Connectivity** section, select the **Publicly accessible** setting to ensure that the reader is accessible from outside your VPC. Note that it is still only accessible through whitelisted IPs. If you are connecting with an SSH tunnel, this can be set to **Not publicly accessible**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-publicly-accessible.png "publicly accessible.png") 6. Click **Add reader**. 7. After a few minutes, the reader status should change to available after it is created. Allow read access to a portion of your Aurora PostgreSQL database or the reader you created in Step 1. ### Configure the security group 1. In your **Amazon RDS** > **Databases** list, click the PostgreSQL instance you want to connect to Prequel. 2. In the database page, in the **Connectivity & security** tab, make note of the **Endpoint** and the **Port** number. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-access.png "reader access.png") 3. If you are connecting directly: in the **Security** section, ensure that set the **Publicly accessible** setting is set to **Yes** to ensure that the destination is accessible from outside your VPC. Note that it is still only accessible through whitelisted IPs. If you are connecting with an SSH tunnel, this can be set to **No**. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-publicly-accessible-yes.png "reader publicly accessible yes.png") 4. Click one of the VPC security groups (usually `default`). Note: VPC groups are permissive (vs. restrictive) and for instances with multiple VPC security groups, only one needs to be configured with the new inbound rule. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-security-groups.png "vpc groups .png") 5. In the **Security Groups** section, select the **Inbound rules** tab. 6. Click **Edit inbound rules** and then click **Add rule**. 7. If you are connecting directly: edit the newly created rule of type **Custom TCP** with the **Port range** noted in the first step (usually `5432`) and a `Custom` **Source** value that includes all of the service IPs. Note: you will need to add `/32` to the end of each IP (CIDR notation). If you are connecting through an SSH tunnel, add the security group of the bastion server, as described in **Step 3** of [AWS SSH Tunneling](/export/sources/additional-options/ssh-tunneling). 8. Click **Save rules**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-add-rule.png "add rule.png") ### Configure network ACLs (access control list) For database instances in a VCP 1. In your RDS dashboard, select the PostgreSQL instance. 2. Click the link to the instance's VPC. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-vpc.png "reader vpc.png") 3. In the VPC menu, click the **VPC ID**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-id.png "vpc id.png") 4. In the **Details** section, click on the link under **Main network ACL**. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-main-network-acl.png "reader main network acl.png") 5. Click on the network ACL ID. ![](https://storage.googleapis.com/prequel_docs/images/postgres-network-acl-id.png "network acl id.png") #### Edit the inbound rules 6. Click on the **Inbound rules** tab, and check if there is an existing rule with a Source of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to **Edit outbound rules**.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-inbound-rules.png "inbound rules.png") 7. Create the inbound rule (if it doesn't exist). Click **Edit inbound rules** and either **Add new rule** or edit an existing rule to allow access to the **port number** of your database instance (usually `5432`) from the Prequel static IP. Click **Save changes**. #### Edit the outbound rules 8. In the ACL menu, select the **Outbound rules** tab, and check if there is an existing rule with a Destination of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to the next step.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-outbound-rules.png "outbound rules.png") 9. Create the outbound rule (if it doesn't exist). Click **Edit outbound rules** and edit the rules to allow outbound traffic to ports 1024-65535 for **Destination** `0.0.0.0/0`. Create a database user to read the source data. 1. Open a connection to your Amazon Aurora PostgreSQL database. 2. Create a user for the data transfer by executing the following SQL command. ```sql title="Create reader user" icon="database" theme={null} CREATE USER PASSWORD '' NOSUPERUSER NOCREATEDB NOCREATEROLE; ``` 3. Grant user read-only access to all relevant schemas and tables. ```sql title="Grant schema access" icon="database" theme={null} GRANT USAGE ON SCHEMA "public" TO ; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ``` 4. Repeat the step above for all schemas with data that needs to be synced. (Replace "public" with the correct schema name) a. **Note**: if you prefer, you may instead grant usage only the desired tables. ```sql title="Grant per-table access" icon="database" expandable theme={null} GRANT USAGE ON SCHEMA "public" TO ; GRANT SELECT ON TO ; GRANT SELECT ON TO ; GRANT SELECT ON TO ; ``` Use the [cURL request](/export/api-reference/sources/create-source) to add the configured source to Prequel. 1. The **name** is a descriptive name of the source for your purposes (i.e., a description) 2. The **host** \[for example, `6.7.8.9` or `your-db.sd8jekhrlkhla.us-east-1.rds.amazonaws.com`] 3. The **port** \[most likely `5432`] 4. The **vendor**: `postgres` 5. The **database**, from Step 2.`postgres` by default, or whatever database you prefer to use 6. The **username** from Step 3 7. The **password** from Step 3 # AWS RDS Source: https://docs.prequel.co/export/sources/postgres-aws-rds Configure an AWS RDS PostgreSQL database as a Prequel source. Follow these steps to configure your source database and connect to Prequel. **Step 1** (creating a read replica) is optional, but will ensure the Prequel service does not put any unnecessary load on your primary database during data transfer. 1. In your Amazon RDS Dashboard, click the PostgreSQL instance you want to replicate. 2. On the database page, click **Actions**, then select **Create read replica** from the drop down. ![](https://storage.googleapis.com/prequel_docs/images/rds-create-read-replica.png "create read replica.png") 3. In the **Instance specifications** sections, specify the instance type for the read replica. It can be smaller than the main instance. ![](https://storage.googleapis.com/prequel_docs/images/rds-instance-class-small.png "instance class small.png") 4. In the **Network & Security** section, under **Publicly accessible** select **Yes** to ensure that the read replica is accessible from outside your VPC. Note that it is still only accessible through whitelisted IPs at this point. ![](https://storage.googleapis.com/prequel_docs/images/rds-publicly-accessible-yes.png "publicly accessible yes.png") 5. In the **Settings** section, enter a **DB instance identifier**. For example, `source-transfer-service-read-replica`. ![](https://storage.googleapis.com/prequel_docs/images/rds-read-replica-identifier.png "read replica settings.png") 6. Click **Create read replica**. 7. The read replica will now start `creating`. Allow write access to a portion of your Aurora PostgreSQL database or the reader you created in Step 1. ### Configure the security group 1. In your **Amazon RDS** > **Databases** list, click the PostgreSQL instance you want to connect to Prequel. 2. In the database page, in the **Connectivity & security** tab, make note of the **Endpoint** and the **Port** number. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-access.png "reader access.png") 3. In the **Security** section, ensure that set the **Publicly accessible** setting is set to **Yes** to ensure that the destination is accessible from outside your VPC. Note that it is still only accessible through whitelisted IPs at this point. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-publicly-accessible-yes.png "reader publicly accessible yes.png") 4. Click one of the VPC security groups (usually `default`). Note: VPC groups are permissive (vs. restrictive) and for instances with multiple VPC security groups, only one needs to be configured with the new inbound rule. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-security-groups.png "vpc groups .png") 5. In the **Security Groups** section, select the **Inbound rules** tab. 6. Click **Edit inbound rules** and then click **Add rule**. 7. Edit the newly created rule of type **Custom TCP** with the **Port range** noted in the first step (usually `5432`) and a `Custom` **Source** value that includes all of the service IPs. Note: you will need to add `/32` to the end of each IP (CIDR notation). 8. Click **Save rules**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-add-rule.png "add rule.png") ### Configure network ACLs (access control list) For database instances in a VCP 1. In your RDS dashboard, select the PostgreSQL instance. 2. Click the link to the instance's VPC. ![](https://storage.googleapis.com/prequel_docs/images/aurora-reader-vpc.png "reader vpc.png") 3. In the VPC menu, click the **VPC ID**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-id.png "vpc id.png") 4. In the **Details** section, click on the link under **Main network ACL**. ![](https://storage.googleapis.com/prequel_docs/images/postgres-main-network-acl-id.png "reader main network acl.png") 5. Click on the network ACL ID. ![](https://storage.googleapis.com/prequel_docs/images/postgres-network-acl-id.png "network acl id.png") #### Edit the inbound rules 6. Click on the **Inbound rules** tab, and check if there is an existing rule with a Source of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to **Edit outbound rules**.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-inbound-rules.png "inbound rules.png") 7. Create the inbound rule (if it doesn't exist). Click **Edit inbound rules** and either **Add new rule** or edit an existing rule to allow access to the **port number** of your database instance (usually `5432`) from the Prequel static IP. Click **Save changes**. #### Edit the outbound rules 8. In the ACL menu, select the **Outbound rules** tab, and check if there is an existing rule with a Destination of `0.0.0.0/0` set to `Allow`. (This is a default rule created by AWS. If this rule already exists, skip to the next step.) ![](https://storage.googleapis.com/prequel_docs/images/postgres-outbound-rules.png "outbound rules.png") 9. Create the outbound rule (if it doesn't exist). Click **Edit outbound rules** and edit the rules to allow outbound traffic to ports 1024-65535 for **Destination** `0.0.0.0/0`. 1. Open a connection to your Amazon RDS PostgreSQL database. 2. Create a user for Prequel by executing the following SQL command. ```sql title="Create read-only user" icon="database" theme={null} CREATE USER PASSWORD '' NOSUPERUSER NOCREATEDB NOCREATEROLE;; ``` Grant user read-only access to all relevant schemas. Though `public` is the default schema, replace with the schema name if needed. ```sql title="Grant schema access" icon="database" theme={null} GRANT USAGE ON SCHEMA "public" TO ; GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO ; ``` 4. Repeat the step above for all schemas with data that needs to be synced. (Replace "public" with the correct schema name) a. **Note**: if you prefer, you may instead grant usage only the desired tables. ```sql title="Grant per-table access" icon="database" expandable theme={null} GRANT USAGE ON SCHEMA "public" TO ; GRANT SELECT ON TO ; GRANT SELECT ON TO ; GRANT SELECT ON TO ; ``` Use the [cURL request](/export/api-reference/sources/create-source) to add the configured source to Prequel. 1. The **name** is a descriptive name of the source for your purposes (i.e., a description) 2. The **host** \[for example, `6.7.8.9` or `your-db.sd8jekhrlkhla.us-east-1.rds.amazonaws.com`] 3. The **port** \[most likely `5432`] 4. The **vendor**: `postgres` 5. The **database**, from Step 2.`postgres` by default, or whatever database you prefer to use 6. The **username** from Step 3 7. The **password** from Step 3 # GCP Source: https://docs.prequel.co/export/sources/postgres-gcp Connect a Postgres (GCP) database as a Prequel source. **Coming soon!** To connect a Postgres (GCP) source, contact support for the setup guide. # Redshift Source: https://docs.prequel.co/export/sources/redshift Instructions for connecting to a Redshift data warehouse as a source 1. In the Redshift console, click **Clusters**. 2. Select the cluster you would like to connect. ![](https://storage.googleapis.com/prequel_docs/images/redshift-cluster.png "redshift cluster.png") 3. In the **General information** pane, make note of the **Endpoint** details. You may need to use the **copy** icon to copy the full details to discover the full endpoint and port number. ![](https://storage.googleapis.com/prequel_docs/images/redshift-endpoint-details.png "redshift endpoint details.png") 1. In the Redshift console, click **Clusters**. 2. Select the cluster you would like to connect. 3. Click the **Properties** tab. 4. Scroll down to the **Network and security settings** section. 5. In the VPC security group field, select a security group to open it. ![](https://storage.googleapis.com/prequel_docs/images/redshift-vpc-security-groups.png "redshift vpc s groups.png") 6. In the Security Groups window, click **Inbound rules**. 7. Click **Edit inbound rules**. 8. In the Edit the Inbound rules window, follow the steps below to create custom TCP rules for Prequel's IP: a. Select **Custom TCP** in the drop-down menu. b. Enter your Redshift port number. (likely `5439`) c. Enter the Prequel static IP address. d. Click **Add rule**. 1. Connect to Redshift using the SQL client. 2. Execute the following query to create a user (replace `` with a password of your choice). ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD ''; ``` 3. Execute the following query to grant the prequel user read-only privileges (replace `` with your database name): ```sql title="Grant read-only access" icon="database" theme={null} GRANT USAGE ON SCHEMA TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ``` Use the UI or a [cURL request](/export/api-reference/overview) to add the configured source to Prequel. 1. The **name** is a descriptive name of the source for your purposes (i.e., a description) 2. The **host** \[for example,`source-redshift.cw1drqwgbdue.us-east-1.redshift.amazonaws.com`] 3. The **port** \[most likely `5439`] 4. The **vendor**: `redshift` 5. The **database** from which tables will be read 6. The **username** from Step 3 7. The **password** from Step 3 # Snowflake Source: https://docs.prequel.co/export/sources/snowflake Instructions for connecting to a Snowflake data warehouse as a source. ## Prerequisites * Your Snowflake user must be granted `securityadmin` and `sysadmin` roles to complete the next step. To verify these roles, run `SHOW GRANTS TO USER ;` and review the `role` column. 1. Review and make any changes to the following setup script. As written, the script should be run once for each table that needs to be transferred. ```sql title="Setup script" icon="database" expandable theme={null} begin; -- create variables for user / password / role / warehouse / database set role_name = 'TRANSFER_ROLE'; -- all letters must be uppercase set user_name = 'TRANSFER_USER'; -- all letters must be uppercase set warehouse_name = 'TRANSFER_WAREHOUSE'; -- all letters must be uppercase set database_name = 'SOME_DATABASE'; set schema_name = 'SOME_SCHEMA'; set table_name = 'SOME_TABLE'; -- repeat for all tables -- change role to securityadmin for user / role steps use role securityadmin; -- create role for data transfer service create role if not exists identifier($role_name); grant role identifier($role_name) to role SYSADMIN; -- create a user for data transfer service create user if not exists identifier($user_name) RSA_PUBLIC_KEY='MIIBIjANBgkqh...'; -- replace with the complete public key copied from the connection form in the onboarding UI -- set default role and warehouse to new user alter user identifier($user_name) SET default_role = $role_name; alter user identifier($user_name) SET default_warehouse = $warehouse_name; alter user identifier($user_name) SET type = service; grant role identifier($role_name) to user identifier($user_name); -- change role to sysadmin for warehouse / database steps use role sysadmin; -- create a warehouse for data transfer service create warehouse if not exists identifier($warehouse_name) warehouse_size = xsmall warehouse_type = standard auto_suspend = 60 auto_resume = true initially_suspended = true; -- grant service role access to warehouse grant USAGE on warehouse identifier($warehouse_name) to role identifier($role_name); -- grant service access to database grant MONITOR, USAGE on database identifier($database_name) to role identifier($role_name); -- use provided database use database identifier($database_name); -- grant service access to schema grant USAGE on schema identifier($schema_name) to role identifier($role_name); -- use provided schema use schema identifier($schema_name); -- grant service access to table grant SELECT on identifier($table_name) to role identifier($role_name); commit; ``` **Using an existing warehouse** By default, this script creates a new warehouse. If you'd prefer to use an existing warehouse, change the `warehouse_name` variable from `TRANSFER_WAREHOUSE` to the name of the warehouse to be shared. 2. In the Snowflake interface, select the **All Queries** checkbox, and click "Run". This will run every query in the script at once. If successful, you will see `Statement executed successfully` in the query results. ![](https://storage.googleapis.com/prequel_docs/images/snowflake-run-all-queries.png "Screen Shot 2021-12-16 at 11.12.34 AM.png") ![](https://storage.googleapis.com/prequel_docs/images/snowflake-statement-executed.png "statement executed.png") By default, the setup script creates a new user using key-pair authentication. If you'd prefer to use username & password authentication, instead of: ```sql title="Key-pair user" icon="database" theme={null} create user if not exists identifier($user_name) RSA_PUBLIC_KEY='MIIBIjANBgkqh...'; ``` Use the following block: ```sql title="Password user" icon="database" theme={null} create user if not exists identifier($user_name) password = 'some_password'; ``` Because service accounts cannot use username & password authentication, you will also need to remove the following line from the provided script: ```sql title="Remove service type" icon="database" theme={null} alter user identifier($user_name) SET type = service; ``` **Using an existing warehouse** By default, this script creates a new warehouse. If you'd prefer to use an existing warehouse, change the `warehouse_name` variable from `TRANSFER_WAREHOUSE` to the name of the warehouse to be shared. After applying the changes above, select the **All Queries** checkbox in the Snowflake interface, and click "Run". This will run every query in the script at once. If successful, you will see `Statement executed successfully` in the query results. ![](https://storage.googleapis.com/prequel_docs/images/snowflake-run-all-queries.png "Screen Shot 2021-12-16 at 11.12.34 AM.png") ![](https://storage.googleapis.com/prequel_docs/images/snowflake-statement-executed.png "statement executed.png") If your Snowflake data warehouse is using Snowflake Access Policies, a new policy must be added to allow the transfer service static IP to write to the warehouse. 1. Review current network policies to check for existing IP safelists. ```sql title="Review network policies" icon="database" theme={null} SHOW NETWORK POLICIES; ``` 2. If there is no existing Snowflake Network Policies (the `SHOW` query returns no results), you can skip to Step 3. 3. If there is an existing Snowflake Network Policy, you must alter the existing policy or create a new one to safelist the data transfer service static IP address. Use the `CREATE NETWORK POLICY` command to specify the IP addresses that can access your Snowflake warehouse. The IP addresses can be found [here](/export/deployment/prequel-ips). ```sql title="Create network policy" icon="database" theme={null} CREATE NETWORK POLICY ALLOWED_IP_LIST = ('5.4.7.8/32'); ``` **Creating your first network policy** If you have no existing network policies and you create your first as part of this step, all other IPs outside of the `ALLOWED_IP_LIST` will be blocked. Snowflake does not allow setting a network policy that blocks your current IP address. (An error message results while trying to create a network policy that blocks the current IP address.) But be careful when setting your first network policy. Use the **host name**, **database name**, **username**, and **password** to complete the connection. # SQL Server Source: https://docs.prequel.co/export/sources/sql-server Configuring your SQL Server source. **Beta source** SQL Server is currently a beta source. Contact the Prequel team to learn more. # Test expression Source: https://docs.prequel.co/import/api-reference/actions/test-expression /generated/openapi-import-generated-2023-12-01.json post /import/actions/test-expr Test an expression. # Generate Import Key Source: https://docs.prequel.co/import/api-reference/auth/generate-import-key /generated/openapi-import-generated-2023-12-01.json post /import/actions/generate-key Generate a public key staged for an import provider. Use the SSH-RSA public key for SSH tunnels and the PKCS#8 public key for Snowflake key pair authentication. # Generate Import Scoped Auth Token Source: https://docs.prequel.co/import/api-reference/auth/generate-import-scoped-auth-token /generated/openapi-import-generated-2023-12-01.json post /import/actions/generate-scoped-auth-token Generate a scoped auth token for a specific import provider. # Generate Provider Import Key Source: https://docs.prequel.co/import/api-reference/auth/generate-provider-import-key /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/actions/generate-key Generate a public key staged for the selected import provider. Use the SSH-RSA public key for SSH tunnels and the PKCS#8 public key for Snowflake key pair authentication. # Generate SSH Key Source: https://docs.prequel.co/import/api-reference/auth/generate-ssh-key /generated/openapi-import-generated-2023-12-01.json post /actions/generate-ssh-key Generate an SSH public key to authenticate into a bastion host when SSH tunneling is enabled. Unused keys are automatically deleted after 30 days. # Create datalake Source: https://docs.prequel.co/import/api-reference/datalakes/create-datalake /generated/openapi-import-generated-2023-12-01.json post /import/datalakes Add a new datalake. # Delete datalake Source: https://docs.prequel.co/import/api-reference/datalakes/delete-datalake /generated/openapi-import-generated-2023-12-01.json delete /import/datalakes/{datalake_id} Delete an existing datalake. This will delete all files in the datalake. # Get datalake Source: https://docs.prequel.co/import/api-reference/datalakes/get-datalake /generated/openapi-import-generated-2023-12-01.json get /import/datalakes/{datalake_id} Retrieve an datalake. # List datalakes Source: https://docs.prequel.co/import/api-reference/datalakes/list-datalakes /generated/openapi-import-generated-2023-12-01.json get /import/datalakes Retrieve list of datalakes. # Query datalake Source: https://docs.prequel.co/import/api-reference/datalakes/query-datalake /generated/openapi-import-generated-2023-12-01.json post /import/datalakes/{datalake_id}/query Execute a query against a datalake. # Update datalake Source: https://docs.prequel.co/import/api-reference/datalakes/update-datalake /generated/openapi-import-generated-2023-12-01.json patch /import/datalakes/{datalake_id} Update an existing datalake. # Create import dataset Source: https://docs.prequel.co/import/api-reference/datasets/create-import-dataset /generated/openapi-import-generated-2023-12-01.json post /import/datasets Add a new import dataset. # Create provider import dataset Source: https://docs.prequel.co/import/api-reference/datasets/create-provider-import-dataset /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/datasets Add a new import dataset for the selected provider. # Delete import dataset Source: https://docs.prequel.co/import/api-reference/datasets/delete-import-dataset /generated/openapi-import-generated-2023-12-01.json delete /import/datasets/{dataset_id} Delete an existing import dataset. # Get import dataset Source: https://docs.prequel.co/import/api-reference/datasets/get-import-dataset /generated/openapi-import-generated-2023-12-01.json get /import/datasets/{dataset_id} Retrieve an import dataset. # Get import dataset record Source: https://docs.prequel.co/import/api-reference/datasets/get-import-dataset-record /generated/openapi-import-generated-2023-12-01.json get /import/datasets/{dataset_id}/records/{record_id} Retrieve a presigned URL for a specific record from the datalake. # Get import dataset schema Source: https://docs.prequel.co/import/api-reference/datasets/get-import-dataset-schema /generated/openapi-import-generated-2023-12-01.json get /import/datasets/{dataset_id}/schema Get the historical schema of an import dataset. # Get provider import dataset Source: https://docs.prequel.co/import/api-reference/datasets/get-provider-import-dataset /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/datasets/{dataset_id} Retrieve an import dataset visible to the selected provider. # Get provider import dataset schema Source: https://docs.prequel.co/import/api-reference/datasets/get-provider-import-dataset-schema /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/datasets/{dataset_id}/schema Get the historical schema of an import dataset visible to the selected provider. # Initialize import extract Source: https://docs.prequel.co/import/api-reference/datasets/initialize-import-extract /generated/openapi-import-generated-2023-12-01.json post /import/datasets/{dataset_id}/extract Manually initialize an extract for an import dataset. # List import datasets Source: https://docs.prequel.co/import/api-reference/datasets/list-import-datasets /generated/openapi-import-generated-2023-12-01.json get /import/datasets Retrieve list of import datasets. # List provider import datasets Source: https://docs.prequel.co/import/api-reference/datasets/list-provider-import-datasets /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/datasets Retrieve import datasets visible to the selected provider. # Random sample dataset Source: https://docs.prequel.co/import/api-reference/datasets/random-sample-dataset /generated/openapi-import-generated-2023-12-01.json post /import/datasets/{dataset_id}/random-sample Get a random sample of records from an import dataset. # Random sample provider dataset Source: https://docs.prequel.co/import/api-reference/datasets/random-sample-provider-dataset /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/datasets/{dataset_id}/random-sample Get a random sample of records from an import dataset visible to the selected provider. # Update import dataset Source: https://docs.prequel.co/import/api-reference/datasets/update-import-dataset /generated/openapi-import-generated-2023-12-01.json patch /import/datasets/{dataset_id} Update an existing import dataset. # Create import destination Source: https://docs.prequel.co/import/api-reference/destinations/create-import-destination /generated/openapi-import-generated-2023-12-01.json post /import/destinations Add a new import destination. # Delete import destination Source: https://docs.prequel.co/import/api-reference/destinations/delete-import-destination /generated/openapi-import-generated-2023-12-01.json delete /import/destinations/{destination_id} Delete an existing import destination. # Get import destination Source: https://docs.prequel.co/import/api-reference/destinations/get-import-destination /generated/openapi-import-generated-2023-12-01.json get /import/destinations/{destination_id} Retrieve an import destination by ID. # Get provider import destination Source: https://docs.prequel.co/import/api-reference/destinations/get-provider-import-destination /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/destinations/{destination_id} Retrieve an import destination visible to the selected provider. # List import destinations Source: https://docs.prequel.co/import/api-reference/destinations/list-import-destinations /generated/openapi-import-generated-2023-12-01.json get /import/destinations Retrieve list of import destinations. # List provider import destinations Source: https://docs.prequel.co/import/api-reference/destinations/list-provider-import-destinations /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/destinations Retrieve import destinations visible to the selected provider. # Update import destination Source: https://docs.prequel.co/import/api-reference/destinations/update-import-destination /generated/openapi-import-generated-2023-12-01.json patch /import/destinations/{destination_id} Update an existing import destination. # Cancel import extract Source: https://docs.prequel.co/import/api-reference/extracts/cancel-import-extract /generated/openapi-import-generated-2023-12-01.json post /import/extracts/{extract_id}/cancel Cancel a running import extract by ID. # Get import extract Source: https://docs.prequel.co/import/api-reference/extracts/get-import-extract /generated/openapi-import-generated-2023-12-01.json get /import/extracts/{extract_id} Get a specific import extract by ID. # Get import extract debug rows Source: https://docs.prequel.co/import/api-reference/extracts/get-import-extract-debug-rows /generated/openapi-import-generated-2023-12-01.json get /import/extracts/{extract_id}/debug Retrieve the rows that were mutated in an extraction. # Get provider import extract Source: https://docs.prequel.co/import/api-reference/extracts/get-provider-import-extract /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/extracts/{extract_id} Get an import extract visible to the selected provider. # Get provider import extract debug rows Source: https://docs.prequel.co/import/api-reference/extracts/get-provider-import-extract-debug-rows /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/extracts/{extract_id}/debug Retrieve debug rows for an import extract visible to the selected provider. # List import extracts Source: https://docs.prequel.co/import/api-reference/extracts/list-import-extracts /generated/openapi-import-generated-2023-12-01.json get /import/extracts List all import extracts for the organization. # List provider import extracts Source: https://docs.prequel.co/import/api-reference/extracts/list-provider-import-extracts /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/extracts List import extracts visible to the selected provider. # API Status Source: https://docs.prequel.co/import/api-reference/health/api-status /generated/openapi-import-generated-2023-12-01.json get / Ping the Prequel API to make sure it is operating. # Delete import dataset Source: https://docs.prequel.co/import/api-reference/import-datasets/delete-import-dataset /generated/openapi-import-generated-2023-12-01.json post /import/datasets/{dataset_id}/delete_records Delete an existing import dataset. # Cancel import load Source: https://docs.prequel.co/import/api-reference/loads/cancel-import-load /generated/openapi-import-generated-2023-12-01.json post /import/loads/{load_id}/cancel Cancel a running import load by ID. # Get import load Source: https://docs.prequel.co/import/api-reference/loads/get-import-load /generated/openapi-import-generated-2023-12-01.json get /import/loads/{load_id} Get a specific import load by ID. # Get import load debug rows Source: https://docs.prequel.co/import/api-reference/loads/get-import-load-debug-rows /generated/openapi-import-generated-2023-12-01.json get /import/loads/{load_id}/debug Retrieve the rows that were mutated in a load. # Get provider import load Source: https://docs.prequel.co/import/api-reference/loads/get-provider-import-load /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/loads/{load_id} Get an import load visible to the selected provider. # List import loads Source: https://docs.prequel.co/import/api-reference/loads/list-import-loads /generated/openapi-import-generated-2023-12-01.json get /import/loads List all import loads for the organization. # List provider import loads Source: https://docs.prequel.co/import/api-reference/loads/list-provider-import-loads /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/loads List import loads visible to the selected provider. # Get log Source: https://docs.prequel.co/import/api-reference/monitoring-&-logging/get-log /generated/openapi-import-generated-2023-12-01.json get /logs/{log_id} Returns the specified audit log. # List logs Source: https://docs.prequel.co/import/api-reference/monitoring-&-logging/list-logs /generated/openapi-import-generated-2023-12-01.json get /logs Returns the audit logs for all events. # Create import product Source: https://docs.prequel.co/import/api-reference/products/create-import-product /generated/openapi-import-generated-2023-12-01.json post /import/products Add a new import product. # Delete import product Source: https://docs.prequel.co/import/api-reference/products/delete-import-product /generated/openapi-import-generated-2023-12-01.json delete /import/products/{product_id} Delete an existing import product. # Get import product Source: https://docs.prequel.co/import/api-reference/products/get-import-product /generated/openapi-import-generated-2023-12-01.json get /import/products/{product_id} Retrieve an import product by ID. # List import products Source: https://docs.prequel.co/import/api-reference/products/list-import-products /generated/openapi-import-generated-2023-12-01.json get /import/products Retrieve list of import products. # Update import product Source: https://docs.prequel.co/import/api-reference/products/update-import-product /generated/openapi-import-generated-2023-12-01.json patch /import/products/{product_id} Update an existing import product. # Create provider Source: https://docs.prequel.co/import/api-reference/providers/create-provider /generated/openapi-import-generated-2023-12-01.json post /import/providers Add a new provider. # Delete provider Source: https://docs.prequel.co/import/api-reference/providers/delete-provider /generated/openapi-import-generated-2023-12-01.json delete /import/providers/{provider_id} Delete an existing provider. # Get provider Source: https://docs.prequel.co/import/api-reference/providers/get-provider /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id} Retrieve a provider. # List providers Source: https://docs.prequel.co/import/api-reference/providers/list-providers /generated/openapi-import-generated-2023-12-01.json get /import/providers Retrieve list of providers. # Update provider Source: https://docs.prequel.co/import/api-reference/providers/update-provider /generated/openapi-import-generated-2023-12-01.json patch /import/providers/{provider_id} Update an existing provider. # Get object storage manifest signature public key Source: https://docs.prequel.co/import/api-reference/signatures/get-object-storage-manifest-signature-public-key /generated/openapi-import-generated-2023-12-01.json get /public/signatures/object-storage-manifest-public-key Get the public key you can use to verify object storage manifests. # Get webhook signature public key Source: https://docs.prequel.co/import/api-reference/signatures/get-webhook-signature-public-key /generated/openapi-import-generated-2023-12-01.json get /public/signatures/webhook-public-key Get the current public key used to verify webhook signatures. Clients should fetch this value instead of hardcoding it because the key may change. # Create import source Source: https://docs.prequel.co/import/api-reference/sources/create-import-source /generated/openapi-import-generated-2023-12-01.json post /import/sources Add a new import source. # Create import source form Source: https://docs.prequel.co/import/api-reference/sources/create-import-source-form /generated/openapi-import-generated-2023-12-01.json post /import/forms/sources Generate fields for an import source form from a partial source definition. # Create provider import source Source: https://docs.prequel.co/import/api-reference/sources/create-provider-import-source /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/sources Add a new import source for the selected provider. # Delete import source Source: https://docs.prequel.co/import/api-reference/sources/delete-import-source /generated/openapi-import-generated-2023-12-01.json delete /import/sources/{source_id} Delete an existing import source. # Delete provider import source Source: https://docs.prequel.co/import/api-reference/sources/delete-provider-import-source /generated/openapi-import-generated-2023-12-01.json delete /import/providers/{provider_id}/sources/{source_id} Delete an import source visible to the selected provider. # Get import source Source: https://docs.prequel.co/import/api-reference/sources/get-import-source /generated/openapi-import-generated-2023-12-01.json get /import/sources/{source_id} Retrieve an import source. # Get import source schema Source: https://docs.prequel.co/import/api-reference/sources/get-import-source-schema /generated/openapi-import-generated-2023-12-01.json post /import/sources/{source_id}/get-schema Get column schema for an import source table or SQL query. # Get provider import source Source: https://docs.prequel.co/import/api-reference/sources/get-provider-import-source /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/sources/{source_id} Retrieve an import source visible to the selected provider. # Get provider import source schema Source: https://docs.prequel.co/import/api-reference/sources/get-provider-import-source-schema /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/sources/{source_id}/get-schema Get column schema for an import source visible to the selected provider. # List import source tables Source: https://docs.prequel.co/import/api-reference/sources/list-import-source-tables /generated/openapi-import-generated-2023-12-01.json post /import/sources/{source_id}/get-tables List tables and columns from an import source by querying information_schema. # List import sources Source: https://docs.prequel.co/import/api-reference/sources/list-import-sources /generated/openapi-import-generated-2023-12-01.json get /import/sources Retrieve list of import sources. # List provider import source tables Source: https://docs.prequel.co/import/api-reference/sources/list-provider-import-source-tables /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/sources/{source_id}/get-tables List tables and columns from an import source visible to the selected provider. # List provider import sources Source: https://docs.prequel.co/import/api-reference/sources/list-provider-import-sources /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/sources Retrieve import sources visible to the selected provider. # Preview import source data Source: https://docs.prequel.co/import/api-reference/sources/preview-import-source-data /generated/openapi-import-generated-2023-12-01.json post /import/sources/{source_id}/preview Preview records from an import source table or SQL query. # Preview provider import source data Source: https://docs.prequel.co/import/api-reference/sources/preview-provider-import-source-data /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/sources/{source_id}/preview Preview records from an import source visible to the selected provider. # Test import source Source: https://docs.prequel.co/import/api-reference/sources/test-import-source /generated/openapi-import-generated-2023-12-01.json post /import/sources/{source_id}/test-source Probe an import source's connectivity and credentials. Optionally accepts a partial 'source' body to preview a credential change without persisting it. # Test new import source Source: https://docs.prequel.co/import/api-reference/sources/test-new-import-source /generated/openapi-import-generated-2023-12-01.json post /import/test-source Validate a candidate (unsaved) import source definition by probing connectivity and credentials. Does not persist the source. # Test new provider import source Source: https://docs.prequel.co/import/api-reference/sources/test-new-provider-import-source /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/test-source Validate a candidate import source definition for the selected provider without persisting it. # Test provider import source Source: https://docs.prequel.co/import/api-reference/sources/test-provider-import-source /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/sources/{source_id}/test-source Probe an import source visible to the selected provider. # Update import source Source: https://docs.prequel.co/import/api-reference/sources/update-import-source /generated/openapi-import-generated-2023-12-01.json patch /import/sources/{source_id} Update an existing import source. # Update provider import source Source: https://docs.prequel.co/import/api-reference/sources/update-provider-import-source /generated/openapi-import-generated-2023-12-01.json patch /import/providers/{provider_id}/sources/{source_id} Update an import source visible to the selected provider. # Create import stream Source: https://docs.prequel.co/import/api-reference/streams/create-import-stream /generated/openapi-import-generated-2023-12-01.json post /import/streams Add a new import stream. # Create provider import stream Source: https://docs.prequel.co/import/api-reference/streams/create-provider-import-stream /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/streams Add a new import stream for the selected provider. # Delete import stream Source: https://docs.prequel.co/import/api-reference/streams/delete-import-stream /generated/openapi-import-generated-2023-12-01.json delete /import/streams/{stream_id} Delete an existing import stream. # Get import stream Source: https://docs.prequel.co/import/api-reference/streams/get-import-stream /generated/openapi-import-generated-2023-12-01.json get /import/streams/{stream_id} Retrieve an import stream by ID. # Get provider import stream Source: https://docs.prequel.co/import/api-reference/streams/get-provider-import-stream /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/streams/{stream_id} Retrieve an import stream visible to the selected provider. # Initialize acknowledgement Source: https://docs.prequel.co/import/api-reference/streams/initialize-acknowledgement /generated/openapi-import-generated-2023-12-01.json post /import/streams/{stream_id}/acknowledge Initialize an acknowledgement job for a stream. # Initialize load Source: https://docs.prequel.co/import/api-reference/streams/initialize-load /generated/openapi-import-generated-2023-12-01.json post /import/streams/{stream_id}/load Initialize a new load for a stream with optional record IDs. # List import streams Source: https://docs.prequel.co/import/api-reference/streams/list-import-streams /generated/openapi-import-generated-2023-12-01.json get /import/streams Retrieve list of import streams. # List provider import streams Source: https://docs.prequel.co/import/api-reference/streams/list-provider-import-streams /generated/openapi-import-generated-2023-12-01.json get /import/providers/{provider_id}/streams Retrieve import streams visible to the selected provider. # Test new provider stream Source: https://docs.prequel.co/import/api-reference/streams/test-new-provider-stream /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/test-stream Validate a candidate import stream for the selected provider without persisting it. # Test new stream Source: https://docs.prequel.co/import/api-reference/streams/test-new-stream /generated/openapi-import-generated-2023-12-01.json post /import/test-stream Validate a candidate (unsaved) stream definition and run its mapping transforms against a sample of extracted records from the named dataset. Does not persist the stream. # Test new stream load Source: https://docs.prequel.co/import/api-reference/streams/test-new-stream-load /generated/openapi-import-generated-2023-12-01.json post /import/test-load Validate a candidate (unsaved) stream definition and run its load mapping against a sample of records. Does not persist the stream. # Test provider stream transform Source: https://docs.prequel.co/import/api-reference/streams/test-provider-stream-transform /generated/openapi-import-generated-2023-12-01.json post /import/providers/{provider_id}/streams/{stream_id}/test-transform Test transforming records for an import stream visible to the selected provider. # Test stream load Source: https://docs.prequel.co/import/api-reference/streams/test-stream-load /generated/openapi-import-generated-2023-12-01.json post /import/streams/{stream_id}/test-load Test loading a predefined set of records for an import stream. This endpoint runs transforms and invokes the destination in 'test mode', returning the outcome without persisting state. # Test stream transform Source: https://docs.prequel.co/import/api-reference/streams/test-stream-transform /generated/openapi-import-generated-2023-12-01.json post /import/streams/{stream_id}/test-transform Test transforming a predefined set of records for an import stream. Optionally accepts a partial 'stream' body to preview the transform output as if the stream were patched (without persisting the patch). # Update import stream Source: https://docs.prequel.co/import/api-reference/streams/update-import-stream /generated/openapi-import-generated-2023-12-01.json patch /import/streams/{stream_id} Update an existing import stream. # Update provider import stream Source: https://docs.prequel.co/import/api-reference/streams/update-provider-import-stream /generated/openapi-import-generated-2023-12-01.json patch /import/providers/{provider_id}/streams/{stream_id} Update an import stream visible to the selected provider. # Delete webhook Source: https://docs.prequel.co/import/api-reference/webhooks/delete-webhook /generated/openapi-import-generated-2023-12-01.json delete /webhooks/{webhook_id} Delete an existing webhook. # Get webhook Source: https://docs.prequel.co/import/api-reference/webhooks/get-webhook /generated/openapi-import-generated-2023-12-01.json get /webhooks/{webhook_id} The endpoint returns the details of a specific webhook by the given ID. # List webhooks Source: https://docs.prequel.co/import/api-reference/webhooks/list-webhooks /generated/openapi-import-generated-2023-12-01.json get /webhooks The webhooks endpoint returns the configured data webhooks. A webhook is an HTTP-based callback function. # Update webhook Source: https://docs.prequel.co/import/api-reference/webhooks/update-webhook /generated/openapi-import-generated-2023-12-01.json patch /webhooks/{webhook_id} Update an existing webhook. # Data types Source: https://docs.prequel.co/import/core-concepts/data-types Understanding how Prequel Import normalizes source data types ## Understanding Prequel data types Because data can be imported from many different source systems, and because data types are not necessarily synonymous across data stores, Prequel Import normalizes every source column into a common Prequel type. The Prequel type determines how each value is represented once it lands in your datalake and how it is delivered to your destination. ## Source type mapping Reference this table to see how each source column type is normalized into a Prequel type. | Prequel Type | BigQuery | Postgres | Redshift | Snowflake | Object storage & SFTP | | ------------ | ---------------------------------- | --------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------- | | `bytes` | `bytes` | `bytea` | `binary varying`, `varbinary`, `varbyte` | `binary`, `varbinary` | `blob` | | `string` | `string` | `char`, `character`, `character varying`, `text`, `uuid`, `varchar` | `bpchar`, `char`, `character`, `character varying`, `nchar`, `nvarchar`, `text`, `varchar` | `char`, `character`, `string`, `text`, `varchar` | `char`, `text`, `uuid`, `varchar` | | `boolean` | `bool`, `boolean` | `bool`, `boolean` | `bool`, `boolean` | `boolean` | `bool`, `boolean` | | `integer` | `int`, `int64`, `smallint` | `int`, `int2`, `int4`, `integer`, `short`, `smallint` | `int`, `int2`, `int4`, `integer`, `smallint`, `tinyint` | `int`, `integer`, `number`, `smallint` | `integer`, `smallint` | | `bigint` | `bigint`, `integer` | `bigint`, `int8` | `bigint`, `int8` | `bigint` | `bigint` | | `decimal` | `bignumeric`, `decimal`, `numeric` | `decimal`, `numeric` | `dec`, `decimal`, `numeric` | `decimal`, `fixed`, `numeric` | `decimal`, `numeric` | | `float` | `float`, `float64` | `double`, `double precision`, `float`, `float4`, `float8`, `real` | `double`, `double precision`, `float`, `float4`, `float8`, `real` | `double`, `double precision`, `float`, `float4`, `float8`, `real` | `double`, `real` | | `timestamp` | `timestamp` | `timestamptz`, `timestamp with time zone` | `timestamptz`, `timestamp with time zone` | `timestamptz`, `timestamp_ltz`, `timestamp_tz` | `timestamptz` | | `date` | `date` | `date` | `date` | `date` | `date` | | `json` | `json` | `json`, `jsonb` | `super` | `object`, `variant` | `json` | | `time` | `time` | `time`, `time without time zone` | `time`, `timetz`, `time with time zone`, `time without time zone` | `time` | `time` | | `geography` | `geography` | `geometry`, `point`, `linestring`, `polygon`, `multipoint`, `multilinestring`, `multipolygon` | `geography`, `geometry`, `point`, `linestring`, `polygon`, `multipoint`, `multilinestring`, `multipolygon` | `geography`, `geometry` | `geometry` | | `vector` | `array` | `array`, `double[]`, `_float4`, `_float8` | `array` | `array` | `double[]` | **Timestamps without a time zone are handled as RFC 3339 strings** Source timestamp types that do not carry a time zone are mapped to the Prequel `string` type rather than `timestamp`. Assuming a time zone such as UTC would be incorrect and could mislead downstream consumers. This applies to: * BigQuery `datetime` * Postgres `timestamp` and `timestamp without time zone` * Redshift `timestamp` and `timestamp without time zone` * Snowflake `datetime`, `timestamp`, and `timestamp_ntz` * Object storage and SFTP `timestamp` These values are preserved as RFC 3339 formatted timestamp strings (e.g., `"2023-05-01T14:30:00"`). To convert these values into actual timestamps, use an [expression](/import/features/table-mapping#expression-syntax) in the Stream's table mapping to parse the string and apply the appropriate time zone. ## Datalake representation Records are stored in your datalake, and delivered to your destination, as JSON. Each Prequel type has a stable JSON representation. | Prequel Type | JSON type | Representation | Example | | ------------ | -------------- | --------------------------- | ---------------------------------------------------- | | `bytes` | String | Base64 encoded string | `"aGVsbG8="` | | `string` | String | UTF-8 string | `"hello"` | | `boolean` | Boolean | JSON boolean | `true` | | `integer` | Number | JSON integer | `42` | | `bigint` | Number | JSON integer | `9223372036854775807` | | `decimal` | String | Numeric string | `"123.456789"` | | `float` | Number | Floating point number | `3.14159` | | `timestamp` | String | RFC 3339 formatted string | `"2023-01-01T14:30:00Z"` | | `date` | String | `YYYY-MM-DD` string | `"2023-01-01"` | | `time` | String | `HH:MM:SS.fffffffff` string | `"14:30:00.000000000"` | | `json` | Object / Array | Embedded JSON | `{"a": 1, "b": [2, 3]}` | | `geography` | Object | GeoJSON object | `{"type": "Point", "coordinates": [-122.41, 37.77]}` | | `vector` | Array | JSON array of numbers | `[0.1, 0.2, 0.3]` | **Decimals are represented as strings to preserve precision** Values of the `decimal` type are serialized as numeric JSON strings rather than JSON numbers. This prevents the loss of precision that would occur if high precision decimals were represented as floating point numbers. **Precision limitations for floating point and integer types** Floating point values are stored in IEEE 754 binary format, which cannot represent all decimal fractions exactly. To ensure accuracy, we **strongly recommend** using fixed precision types such as `decimal` instead. Integer values must fit within a signed 64-bit range. Values above `9223372036854775807` will overflow. # Dataflow Source: https://docs.prequel.co/import/core-concepts/dataflow Understand how data flows through Prequel Import from source to destination ## Lifecycle of a transfer ![Lifecycle of a transfer](https://storage.googleapis.com/prequel_docs/images/import-lifecycle-of-a-transfer.png) Data flows through Prequel Import in roughly the following sequence: 1. **[Source](/import/core-concepts/sources)** - The origin of your provider's data (e.g., a database, data warehouse, object storage, or other data platform) 2. **[Dataset](/import/core-concepts/datasets)** - The cached table or query for the data to be synced 3. **[Stream](/import/core-concepts/streams)** - The loading pipeline that batches, maps, and delivers data to the target destination 4. **[Destination](/import/core-concepts/destinations)** - The target endpoint where processed data is delivered (API, with Kafka topic coming soon) ## Providers Prequel Import is built with multi-tenancy in mind. Sources, Datasets, and Streams must all be associated with a Provider. To learn more about multi-tenancy and the Provider concept, see the [Provider](/import/core-concepts/providers) guide. ![Multi-tenancy dataflow](https://storage.googleapis.com/prequel_docs/images/import-multitenancy-dataflow.png) # Datalake Source: https://docs.prequel.co/import/core-concepts/datalake Understand how Prequel Import uses your datalake for secure data caching and transfer ## What is the datalake? Prequel Import makes use of a secure cache of the source data in your cloud storage bucket. This enables efficient change detection without the need to store any customer data on Prequel's core infrastructure. ## How is the datalake structured? A Datalake configures one cloud storage bucket. Each Provider is assigned to exactly one Datalake at creation, and that assignment is immutable. The datalake can be single-tenant or multi-tenant: * **Shared Datalake:** Many Providers point to the same Datalake. Prequel partitions the shared bucket internally by tenant, so data from separate Providers is never comingled. * **Per-Provider Datalake:** Each Provider has a dedicated Datalake. Use this if you prefer bucket-level isolation. You can mix the two patterns within a single environment, sharing Datalakes among some customers and pinning other customers to their own. ## Lifecycle responsibility Prequel manages lifecycle for cached data under the `lakehouse/` prefix of your datalake. Staging data written to the `artifacts/` prefix is not managed by Prequel. You must configure a lifecycle policy on the `artifacts/` prefix (or `/artifacts/` if you set a bucket prefix on the datalake) to expire it. The setup steps below include a lifecycle policy step for each vendor. We recommend matching the lifecycle policy duration to your datalake's `retention_window_days`. ## Configuring your datalake In the AWS console, navigate to the **S3** service page and click **Create bucket**. Enter a **Bucket name** and choose an **AWS Region**. We recommend setting **Object Ownership** to "ACLs disabled" and **Block Public Access settings for this bucket** to "Block all public access". Make a note of the bucket name and region. Create an IAM policy with the following permissions, replacing `BUCKET_NAME` with the name of the bucket you created: ```json title="IAM policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` Create an IAM role with the trust policy below, then attach the policy from the previous step. Replace `` with the service account ID from your deployment details. ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "accounts.google.com" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` Once the role is created, make a note of its **ARN**. On the bucket, navigate to **Management** > **Lifecycle rules** and click **Create lifecycle rule**. Limit the rule scope by prefix to `artifacts/` (or `/artifacts/` if you set a bucket prefix on the datalake), then add an **Expire current versions of objects** action set to the same number of days as the datalake's `retention_window_days`. In the GCP console, navigate to **Cloud Storage** and click **Create**. Enter a **bucket name** and choose a **region**. At the **Choose how to control access to objects** step, we recommend selecting **Enforce public access prevention on this bucket**. Make a note of the bucket name and region. Navigate to **IAM & Admin** > **Service Accounts** and click **Create service account**. Name the service account and click **Done**. You will grant permissions in the next steps. On the bucket you created, grant the service account the following permissions, either via a custom role or via the built-in **Storage Object Admin** role plus `storage.buckets.get`: * `storage.buckets.get` * `storage.objects.list` * `storage.objects.get` * `storage.objects.create` * `storage.objects.delete` Open the service account you created, select the **Permissions** tab, and click **Grant access**. Add the Prequel service account from your deployment details as a principal and assign the **Service Account Token Creator** role. Click **Save**. This allows Prequel to impersonate your service account via short-lived tokens. Prequel never handles your service account keys, and all operations appear in your GCP audit logs. On the bucket, open the **Lifecycle** tab and click **Add a rule**. Set the action to **Delete object**, with the conditions **Object name matches prefix** = `artifacts/` (or `/artifacts/` if you set a bucket prefix on the datalake) and **Age** equal to the datalake's `retention_window_days`. Use the following details to complete the datalake setup: **bucket name**, **bucket region**, and the **service account email** you created. In the Azure portal, navigate to **Storage accounts** and select an existing account or create a new one. In the account's **Configuration** settings, ensure **Storage account key access** is enabled and **Hierarchical namespace** is enabled. In the storage account, navigate to **Containers** under **Data storage** and click **+ Container**. Choose a name and click **Create**. In the storage account, navigate to **Security + networking** > **Access keys**. Reveal one of the two keys and make a note of it. This key grants full access to the storage account, so store it securely. In the storage account, navigate to **Data management** > **Lifecycle management** and click **Add a rule**. Scope the rule to **Limit blobs with filters** with the prefix `/artifacts/` (or `//artifacts/` if you set a bucket prefix on the datalake), and set **Delete the blob** to the same number of days as the datalake's `retention_window_days`. Use the following details to complete the datalake setup: **storage account name**, **container name**, and the **storage account access key**. # Datasets Source: https://docs.prequel.co/import/core-concepts/datasets Learn how and why Datasets are replicated from Sources In Prequel Import, a Dataset is a materialized cache of data representing a connected Source table or custom SQL query. Datasets are maintained in the [Datalake](/import/core-concepts/datalake). Every Dataset is defined along two independent dimensions: its **type**, which controls how Prequel Import detects changes, and its **method**, which controls how Prequel Import reads data from the Source. **Dimension** or **Fact**. Determines how Prequel Import detects changes. **Table**, **SQL**, or **Glob**. Determines how Prequel Import reads from the Source. ## 1. Dataset type A Dataset's type determines which columns Prequel Import uses to detect changes and whether it can sync incrementally. It applies the same way to every method. See [Change Detection](/import/features/change-detection) for how each type is synced. **Dimension tables** contain rows that may be updated over time. These typically represent **entities** such as users, products, or accounts. Prequel Import tracks changes and syncs updates to existing records. **Configuration fields** See the [API reference](/import/api-reference/datasets/create-import-dataset) for the full payload. The column used to uniquely identify records. Must be a non-null string on every row. See [Dataset requirements](#dataset-requirements). The column Prequel Import uses to find rows changed since the last sync. When set, every sync after the first reads only rows modified since the previous run. When unset, every source row is read on each sync. Either way records are merged by `primary_key_column`, so this setting controls whether the source read is filtered by time, not whether rows are replaced or removed. A boolean column the Source uses to flag deleted rows. Prequel Import reads this flag and propagates those deletions to the Datalake. Rows that are physically removed from the Source are not detected, whether or not this column is set. **Fact tables** are an append-only stream of **immutable events or transactions**, such as events, page views, or logs. New records are added but existing records are never modified. **Configuration fields** See the [API reference](/import/api-reference/datasets/create-import-dataset) for the full payload. The column used to uniquely identify records. Must be a non-null string on every row. See [Dataset requirements](#dataset-requirements). The column recording when each record was created. Required for efficient append-only change detection. ## 2. Dataset method A Dataset's method determines how Prequel Import reads data from the Source. Reads a named table from the Source, addressed by namespace and table name. **Configuration fields** See the [API reference](/import/api-reference/datasets/create-import-dataset) for the full payload. The namespace (for example schema or database) containing the source table. The name of the table in the Source. Reads the result set of a custom `SELECT` query you provide, so you can join, filter, or reshape data before it is replicated. **Configuration fields** See the [API reference](/import/api-reference/datasets/create-import-dataset) for the full payload. The read-only `SELECT` statement Prequel Import runs against the Source. Prequel Import reads your query as a subquery, so a few constraints apply: * The query must return a result set. Use a `SELECT` or another read-only statement that returns rows. Statements that do not return rows, such as `INSERT` or `UPDATE`, are not supported. * Every column named in your type configuration (`primary_key_column`, `last_modified_column`, `is_deleted_column`, `created_at_column`) must appear in the query's result set. If a configured column is missing, the extract fails. For an incremental Dimension Dataset, the `last_modified_column` value must be stable and monotonically increasing so change detection advances correctly. Computed or non-monotonic timestamps can cause rows to be missed or re-read. ```json title="Dataset config" icon="brackets-curly" expandable theme={null} { "dataset": { "source_id": "1f3c9e7a-0b2d-4a8e-9c11-7d6f2b5a4c33", "method": "sql", "type": "dimension", "frequency_minutes": 60, "sql": { "query": "SELECT p.id, p.name, p.updated_at, c.label AS category FROM products p JOIN categories c ON c.id = p.category_id WHERE p.active = true" }, "dimension": { "primary_key_column": "id", "last_modified_column": "updated_at" } } } ``` Reads object files that match a pattern. Available for object storage Sources. **Configuration fields** See the [API reference](/import/api-reference/datasets/create-import-dataset) for the full payload. The glob pattern matching object files in the Source. The format of the object files. One of `csv`, `parquet`, `jsonl`, `json`. **Virtual columns for objects** Objects dropped into object storage often carry no row-level timestamps, which would force every sync to re-read every object. To solve this, each extracted record includes two virtual columns derived from the object's storage metadata: * `meta__filename` (`string`): the key or path of the object the record was read from * `meta__file_last_modified` (`timestamp`): the object's last modified time, as recorded by the storage provider Set `last_modified_column` to `meta__file_last_modified` to sync incrementally at the object level: each sync reads only objects added or modified since the previous run. Note the granularity: if any part of an object changes, every record in that object is re-read. These columns are computed during extraction and are never written to your Source. They can also be referenced in [Stream mappings](/import/features/table-mapping#object-storage-virtual-columns). ## Dataset requirements Every record Prequel Import reads from the Source must meet these requirements, whatever the Dataset's type or method. * **A primary key on every record.** The column named by `primary_key_column` must be present and non-null on every row the Dataset returns. * **A string primary key.** The `primary_key_column` must be a string type in the Source, such as `text` or `varchar`. Numeric and UUID columns need to be cast to a string, which the SQL method or a view in the Source can do. * **Records under 32MB.** The total size of each record must remain under 32MB to be processed. ## Creating a dataset A Dataset is created with `POST /import/datasets`. Along with one type block and one method block from above, every Dataset specifies these common fields. **Configuration fields** See the [API reference](/import/api-reference/datasets/create-import-dataset) for the full payload. A name for the Dataset. The ID of the [Source](/import/core-concepts/sources) this Dataset reads from. How Prequel Import reads from the Source. One of `table`, `sql`, `glob`. The type of Dataset. One of `dimension`, `fact`. How often (in minutes) Prequel Import checks this Dataset for changes. Must be at least 1. Whether the Dataset is enabled. Defaults to `true` when omitted. ## Updating a dataset After a Dataset is created, only its `name`, `frequency_minutes`, and `is_enabled` can be changed, with [`PATCH /import/datasets/{id}`](/import/api-reference/datasets/update-import-dataset). The type, method, query, and column configuration are fixed when the Dataset is created. To change a query or column mapping, create a new Dataset. # Destinations Source: https://docs.prequel.co/import/core-concepts/destinations Where Prequel Import delivers your data. A destination is an endpoint you configure in Prequel to receive imported records. Prequel supports multiple destination types depending on your delivery needs. Every destination requires a `record_schema`: a JSON Schema draft-07 document that defines the shape of records Prequel will deliver. This defines the contract between Prequel and your endpoint. ## Destination types | Type | Best for | Description | | ------------------------------------------------------------ | ------------------------- | ------------------------------------------------------ | | [`webhook_batch`](/import/destination-specs/webhook-batch) | High volume (recommended) | Delivers multiple records per request. | | [`webhook_record`](/import/destination-specs/webhook-record) | Lower volume | Delivers one record per request. Simpler to implement. | | [`kafka_topic`](/import/destination-specs/kafka-topic) | Streaming (coming soon) | Delivers records directly to a Kafka topic. | For the full destination configuration reference and a complete example, see [Create Destination](/import/import-api/overview). # Providers Source: https://docs.prequel.co/import/core-concepts/providers Understand how Prequel Import handles multi-tenancy with Providers ## Provider associations on Prequel objects Providers represent unique tenants in your platform. Providers may have one or many Sources, Datasets, Streams, and can map to one or many Destinations. ![Multi-tenancy dataflow](https://storage.googleapis.com/prequel_docs/images/import-multitenancy-dataflow.png) ## Provider configuration | Field | Type | Required | Description | | ------------- | -------- | -------- | --------------------------------------------------------------------------------------------------- | | `id` | `string` | Yes | A user-provided identifier for this Provider. Immutable after creation. | | `name` | `string` | Yes | A human-readable name for this Provider. | | `datalake_id` | `uuid` | Yes | The ID of the [Datalake](/import/core-concepts/datalake) where this Provider's data will be staged. | | `products` | `array` | Yes | One or more product names this Provider has access to. At least one is required. | ## Provider-scoped API endpoints For embedded experiences, Prequel offers SDK and API endpoints that can (or must) be scoped by Provider. # Sources Source: https://docs.prequel.co/import/core-concepts/sources Where Prequel Import reads your providers' data from. A source is the system Prequel Import reads a provider's data from: a data warehouse, relational database, or object storage location. Every source belongs to a [Provider](/import/core-concepts/providers), and Prequel regularly detects changes in the source and delivers them to your configured [Destination](/import/core-concepts/destinations). For the full list of supported source types and vendor-specific setup guides, see the [Sources overview](/import/sources/overview). # Streams Source: https://docs.prequel.co/import/core-concepts/streams Understand how Streams transform, map, and orchestrate data loading into the Destination. A Stream connects a Dataset to a Destination and defines how records are mapped and delivered. Prequel Import orchestrates the Stream, resiliently batching, transforming, and delivering records to the configured Destination. ## Stream configuration options | Field | Type | Required | Description | | ---------------- | --------- | -------- | --------------------------------------------------------------------------------------------- | | `name` | `string` | Yes | A descriptive name for the Stream. | | `destination_id` | `uuid` | Yes | The ID of the [Destination](/import/core-concepts/destinations) where data will be delivered. | | `dataset_id` | `uuid` | Yes | The ID of the [Dataset](/import/core-concepts/datasets) this Stream reads from. | | `mappings` | `array` | Yes | An array of field mappings that transform source fields to target fields at the Destination. | | `is_enabled` | `boolean` | No | Whether the Stream is actively delivering. Defaults to enabled. | ## Mappings Each mapping defines how a source field maps to a target field in the Destination. Mappings can either reference a Source column directly or compute a value from one or more Source columns using an expression. See [Advanced Table Mapping](/import/features/table-mapping) for the two mapping categories, expression syntax, and additional supported functions. # Kafka topic Source: https://docs.prequel.co/import/destination-specs/kafka-topic Configure a Kafka topic to receive data from Prequel Import A `kafka_topic` destination delivers records directly to a Kafka topic. Prequel Import behaves as a Producer and emits one message per record. To configure a Kafka destination, you must implement a spec-compliant Kafka topic that conforms to the following specification. ## Kafka topic specification The Kafka topic specification is under development. For early access, please reach out to your Prequel Representative. # Batch API Source: https://docs.prequel.co/import/destination-specs/webhook-batch Implementing a Prequel Destination for your batch-based receiving endpoint A Batch-based API endpoint is the recommended implementation for most use cases. Contact the Prequel team if you have questions. A `webhook_batch` destination delivers multiple records in a single HTTP request. For the API call to register a destination with Prequel, including a complete example, see [Create Destination](/import/import-api/overview). Declare your schema and request template within Prequel. Receive, download, and process the batch file. Return a status code Prequel acts on. Optionally, report the outcome of each record. ## Authentication Prequel signs every delivery request with the `X-Prequel-Webhook-Signature` header so your endpoint can validate the request originated from Prequel. Fetch the public key for verification from `GET /public/signatures/webhook-public-key`. See [Webhook headers](#webhook-headers) for the full header list and verification steps. After verifying the request signature, you can optionally read `provider_id` and `load_id` from the envelope body to confirm the request belongs to an expected provider and load. ## Creating your destination spec The `destination` object allows you to customize the contract between data shared by your customers and the shape Prequel provides to your receiving endpoint. **Example:** Suppose you are importing a `users` table with string fields `id`, `email`, and `subscription`. Below are four possible shapes for a `webhook_batch` destination. These are not exhaustive and features can be combined to confirm to your endpoint's requirements: 1. **Schema as body:** Records land in the `JSONL` artifact in the shape declared by `record_schema`, with no additional shaping. 2. **Custom body:** Use a `body` template to customize each record. Only supported when `format` is `json`. 3. **Routed by provider:** `uri` and/or `headers` vary at delivery time using the `ProviderID` delivery-level identifier. 4. **Destination per provider:** A separate destination is registered for each provider, with a `record_schema` that includes provider-specific custom fields. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "users-batch-destination", "type": "webhook_batch", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" } }, "required": ["id", "email"] }, "webhook_batch": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/batch", "headers": { "Content-Type": "application/json" } }, "format": "json" } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/batch Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "url": "https://storage.example.com/batches/f47ac10b.jsonl?X-Signature=...", "batch_acknowledgement_url": "https://storage.example.com/acks/f47ac10b.json?X-Signature=...", "format": "json", "record_count": 2500, "batch_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } ``` ```json title="File at the presigned url" expandable theme={null} { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro", "prequel__is_deleted": false, "prequel__record_id": "usr_001" } { "id": "usr_002", "email": "raj@acme.com", "subscription": "free", "prequel__is_deleted": false, "prequel__record_id": "usr_002" } ``` Set `name` (string, required) to a unique identifier for this destination within your account. Set `type` (string, required) to `webhook_batch` for batch deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery. Set `webhook_batch.request_template` (object, required) with the HTTP `method` (string, defaults to `POST`), `uri` (string, required), and optional `headers` (object). The `uri` and `headers` templates support the `.Prequel.*` namespace. Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces. Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "users-batch-destination", "type": "webhook_batch", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" } }, "required": ["id", "email"] }, "webhook_batch": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/batch", "headers": { "Content-Type": "application/json" }, "body": "{\"email\": \"{{.Record.email}}\", \"user\": {{.Record.AsJson}}, \"is_deleted\": {{.IsDeleted}}, \"provider_id\": \"{{.Prequel.ProviderID}}\", \"load_id\": \"{{.Prequel.LoadID}}\"}" }, "format": "json" } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/batch Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "url": "https://storage.example.com/batches/f47ac10b.jsonl?X-Signature=...", "batch_acknowledgement_url": "https://storage.example.com/acks/f47ac10b.json?X-Signature=...", "format": "json", "record_count": 2500, "batch_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } ``` ```json title="File at the presigned url" expandable theme={null} { "email": "jane@acme.com", "user": { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro" }, "is_deleted": false, "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } { "email": "raj@acme.com", "user": { "id": "usr_002", "email": "raj@acme.com", "subscription": "free" }, "is_deleted": false, "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } ``` Set `name` (string, required) to a unique identifier for this destination within your account. Set `type` (string, required) to `webhook_batch` for batch deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery. Set `webhook_batch.request_template` (object, required) with `method`, `uri`, `headers`, and `body`. The `body` template renders once per record while Prequel encodes the `JSONL` artifact, with access to record fields, the deletion flag, and delivery-level identifiers. The `uri` and `headers` templates render once per delivery and have access to delivery-level identifiers only. See [Template variables](#template-variables) for the full list of available variables. Body templates are only valid when `format` is `json`; selecting `csv` or `parquet` rejects the destination. Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces. Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "users-batch-destination", "type": "webhook_batch", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" } }, "required": ["id", "email"] }, "webhook_batch": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/batch?provider={{.Prequel.ProviderID}}", "headers": { "Content-Type": "application/json" } }, "format": "json" } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/batch?provider=b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63 Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "url": "https://storage.example.com/batches/f47ac10b.jsonl?X-Signature=...", "batch_acknowledgement_url": "https://storage.example.com/acks/f47ac10b.json?X-Signature=...", "format": "json", "record_count": 2500, "batch_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } ``` ```json title="File at the presigned url" expandable theme={null} { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro", "prequel__is_deleted": false, "prequel__record_id": "usr_001" } { "id": "usr_002", "email": "raj@acme.com", "subscription": "free", "prequel__is_deleted": false, "prequel__record_id": "usr_002" } ``` Set `name` (string, required) to a unique identifier for this destination within your account. Set `type` (string, required) to `webhook_batch` for batch deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery. Set `webhook_batch.request_template` (object, required) with `method`, `uri`, and optional `headers`. Use `{{.Prequel.ProviderID}}` in `uri` and `headers` to route by the provider whose data is being delivered. These templates render once per delivery, before any records are selected. See [Template variables](#template-variables) for the full list of identifiers available. Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces. Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "acme-corp-batch-destination", "type": "webhook_batch", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" }, "tax_id": { "type": "string" } }, "required": ["id", "email", "tax_id"] }, "webhook_batch": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/batch/acme-corp", "headers": { "Content-Type": "application/json" } }, "format": "json" } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/batch/acme-corp Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "url": "https://storage.example.com/batches/f47ac10b.jsonl?X-Signature=...", "batch_acknowledgement_url": "https://storage.example.com/acks/f47ac10b.json?X-Signature=...", "format": "json", "record_count": 312, "batch_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } ``` ```json title="File at the presigned url" expandable theme={null} { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro", "tax_id": "US-77-1234567", "prequel__is_deleted": false, "prequel__record_id": "usr_001" } { "id": "usr_002", "email": "raj@acme.com", "subscription": "free", "tax_id": "US-77-7654321", "prequel__is_deleted": false, "prequel__record_id": "usr_002" } ``` Create a separate destination for each provider whose requirements differ. Give each one a `name` (string, required) that identifies the provider it serves. Set `type` (string, required) to `webhook_batch` for batch deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields this provider sends, including any custom fields unique to them. Prequel validates records against this schema before delivery. Set `webhook_batch.request_template` (object, required) with `method`, `uri`, and optional `headers`. Because the destination is scoped to a single provider, you can hard-code provider-specific routing into `uri` and `headers` without needing template variables. Set `webhook_batch.format` (string, required) to one of `json`, `csv`, or `parquet`. See [Delivery format](#delivery-format) for what each option produces. Optionally set `webhook_batch.max_size_per_batch` (integer, default `24000`), `webhook_batch.max_batches_per_minute` (integer, default `10`), and `webhook_batch.max_concurrency` (integer, default `1`) to control delivery throughput. ## Template variables The `body` template has access to every variable below; the `uri` and `headers` templates only have access to the `.Prequel.*` namespace. | Variable | Type | Description | | ---------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------- | | `{{.Record.}}` | `any` | Value of a specific field from the record (e.g. `{{.Record.email}}`). The field must be declared in `record_schema`. | | `{{.Record.AsJson}}` | `object` | The full record serialized as a JSON object. | | `{{.IsDeleted}}` | `boolean` | `true` when Prequel is delivering a record deletion, `false` otherwise. | | `{{.Prequel.ProviderID}}` | `string` | Identifier for the provider whose data is being delivered. | | `{{.Prequel.LoadID}}` | `string` | Identifier for the active load job. | | `{{.Prequel.DestinationID}}` | `string` | Identifier for the destination receiving this delivery. | | `{{.Prequel.StreamID}}` | `string` | Identifier for the stream this delivery came from. | | `{{.Prequel.TenantID}}` | `string` | Identifier for the tenant the stream belongs to. | | `{{.Prequel.IsTest}}` | `boolean` | `true` when the delivery comes from a test load. | ## Delivery format Prequel makes one HTTP request per batch. The request body is an envelope with the following fields: | Field | Type | Description | | --------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `url` | `string` | Presigned URL to the batch file. The URL itself carries a 24-hour TTL set by object storage. | | `batch_acknowledgement_url` | `string` | URL to `PUT` acknowledgements for this batch to, expiring 24 hours after the delivery. See [Acknowledgements](/import/features/acknowledgements). | | `format` | `string` | Format of the file at `url`. One of `json`, `csv`, or `parquet`. | | `record_count` | `integer` | Number of records contained in the batch file. | | `batch_id` | `string` | Unique identifier for this batch. | | `provider_id` | `string` | Identifier for the provider whose data is being delivered. Omitted when unavailable. | | `load_id` | `string` | Identifier for the active load job. Omitted when unavailable. | Your endpoint reads the envelope, downloads the file at `url`, and processes the records inside. The `format` field on the destination controls what kind of file the URL points to: | Format | File at `url` | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `json` | `JSONL`. Each line is one record, optionally shaped by the `body` template. Without a `body` template, each line also carries `prequel__is_deleted` and `prequel__record_id` metadata fields. | | `csv` | `CSV`. Includes `prequel__is_deleted` and `prequel__record_id` columns. Body templates are not supported. | | `parquet` | `Parquet`. Includes `prequel__is_deleted` and `prequel__record_id` columns. Body templates are not supported. | Without a `body` template, each delivered record also carries two Prequel-managed fields: `prequel__record_id`, a stable identifier for the record, and `prequel__is_deleted`, which is `true` when the record represents a deletion. ## Webhook headers Prequel includes the following headers on every delivery request. | Header | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X-Prequel-Webhook-Signature` | SHA-256 RSA PKCS1 v1.5 signature of the payload. To verify, fetch the public key from `GET /public/signatures/webhook-public-key` and check the hex-decoded signature against the SHA-256 of `.` using PKCS1 v1.5. | | `X-Prequel-Webhook-Timestamp` | RFC 3339 timestamp of when the request was sent. Reject stale timestamps to prevent replay attacks. | | `X-Prequel-Webhook-Digest` | SHA-256 hash of the raw request body. | | `Content-Type` | For batch deliveries, defaults to `application/json` unless overridden by the `headers` template. Record deliveries send only the headers the `headers` template specifies. | | `X-Prequel-Is-Test` | Sent with value `true` only on deliveries from a test load. Acknowledgements uploaded for test deliveries are not ingested. | | `X-Prequel-Idempotency-Key` | Unique idempotency key across retry attempts of the same batch or record within a load job. | | `Content-Type` | Defaults to `application/json` unless overridden by the `headers` template. | ## Response codes Your endpoint should return the following response codes. | Code | When to return it | What Prequel does | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `2xx` | You accepted the record. | Records the delivery as successful. | | `429`, `503`, `504` with a `Retry-After` header | You are rate limited or temporarily unavailable and want the delivery attempted again. | Waits the interval the header specifies, capped at 30 seconds, then retries, for a maximum of three attempts per delivery. | | `400`, `412`, `413`, `422` | The payload itself is the problem. Return `422` for schema or data validation failures, `413` when it exceeds your size limit, and `400` when the request is malformed. | Records the delivery as rejected by your endpoint. Does not retry. | | Any other status | Your system could not accept the delivery. | Records the delivery as failed at the destination. This includes `500` and `502`, and a `429`, `503`, or `504` whose `Retry-After` header is absent or unparseable. Does not retry. | For more detail on this retry behavior see: Retry rules, redelivery across loads, and the circuit breakers that stop a failing stream. ## Error codes Prequel records the [error code](/import/logging/error-codes) below against every record in the batch, since the response describes the request rather than any individual record. | Status | Error code | Title | | ----------- | ----------- | ---------------------------------------- | | `400` | `PRQL-JPDQ` | Import Destination Bad Request | | `403` | `PRQL-CNJK` | Import Destination Forbidden | | `404` | `PRQL-VFTB` | Import Destination Not Found | | `413` | `PRQL-QAHC` | Import Destination Payload Too Large | | `422` | `PRQL-NROT` | Import Destination Unprocessable Entity | | `429` | `PRQL-WQKF` | Import Destination Rate Limited | | `431` | `PRQL-QFJW` | Import Destination Headers Too Large | | `500` | `PRQL-PHOK` | Import Destination Internal Server Error | | `502` | `PRQL-QPMF` | Import Destination Bad Gateway | | `503` | `PRQL-HNNX` | Import Destination Service Unavailable | | `504` | `PRQL-PRHQ` | Import Destination Gateway Timeout | | Other `4xx` | `PRQL-IMEK` | Import Destination Client Error | | Other `5xx` | `PRQL-TNXG` | Import Destination Server Error | ## Acknowledgements Without acknowledgements, all records within a batch are forced to accept the same status from your webhook response. Acknowledging is what makes individually assigned, record-level failures visible to you and your customer to address. Report each record's downstream outcome back to Prequel once your system has processed it. # Record API Source: https://docs.prequel.co/import/destination-specs/webhook-record Implementing a Prequel Destination for your record-based receiving endpoint A `webhook_record` destination delivers one HTTP request per record. For the API call to register a destination with Prequel, including a complete example, see [Create Destination](/import/import-api/overview). For higher-volume workloads, consider the [Batch API](/import/destination-specs/webhook-batch), which delivers multiple records per request. Declare your schema and request template within Prequel. Receive and process the record in the request body. Return a status code Prequel acts on. Optionally, report the outcome of each record. ## Authentication Prequel signs every delivery request with the `X-Prequel-Webhook-Signature` header so your endpoint can validate the request originated from Prequel. Fetch the public key for verification from `GET /public/signatures/webhook-public-key`. See [Webhook headers](#webhook-headers) for the full header list and verification steps. ## Creating your destination spec The `destination` object allows you to customize the contract between data shared by your customers and the shape Prequel provides to your receiving endpoint. **Example:** Suppose you are importing a `users` table with string fields `id`, `email`, and `subscription`. Below are four possible shapes for a `webhook_record` destination. These are not exhaustive and features can be combined to conform to your endpoint's requirements: 1. **Schema as body:** Each request body is the record in the shape declared by `record_schema`, with no additional shaping. 2. **Custom body:** Use a `body` template to customize each record. 3. **Routed by provider:** `uri` and/or `headers` vary at delivery time using the `ProviderID` delivery-level identifier. 4. **Destination per provider:** A separate destination is registered for each provider, with a `record_schema` that includes provider-specific custom fields. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "users-record-destination", "type": "webhook_record", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" } }, "required": ["id", "email"] }, "webhook_record": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/records", "headers": { "Content-Type": "application/json" } } } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/records Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro" } ``` Set `name` (string, required) to a unique identifier for this destination within your account. Set `type` (string, required) to `webhook_record` for record deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery. Set `webhook_record.request_template` (object, required) with the HTTP `method` (string, defaults to `POST`), `uri` (string, required), and optional `headers` (object). The `uri` and `headers` templates support the `.Prequel.*` namespace, `.Record.*` fields, and `.IsDeleted`. With no `body` template, Prequel sends each record as a JSON object. Optionally set `webhook_record.max_records_per_minute` (integer, default `3000`) and `webhook_record.max_concurrency` (integer, default `1`) to control delivery throughput. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "users-record-destination", "type": "webhook_record", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" } }, "required": ["id", "email"] }, "webhook_record": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/records", "headers": { "Content-Type": "application/json" }, "body": "{\"email\": \"{{.Record.email}}\", \"user\": {{.Record.AsJson}}, \"is_deleted\": {{.IsDeleted}}, \"provider_id\": \"{{.Prequel.ProviderID}}\", \"load_id\": \"{{.Prequel.LoadID}}\"}" } } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/records Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "email": "jane@acme.com", "user": { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro" }, "is_deleted": false, "provider_id": "b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63", "load_id": "7e0c2b9a-5d14-4f6e-8a3b-9c47d0e15f28" } ``` Set `name` (string, required) to a unique identifier for this destination within your account. Set `type` (string, required) to `webhook_record` for record deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery. Set `webhook_record.request_template` (object, required) with `method`, `uri`, `headers`, and `body`. The `body` template renders once per request and becomes the HTTP body, with access to record fields, the deletion flag, and delivery-level identifiers. The `uri` and `headers` templates have access to the same variables, scoped to `.Prequel.*`, `.Record.*`, and `.IsDeleted`. See [Template variables](#template-variables) for the full list of available variables. To validate delivery context, render values like `{{.Prequel.ProviderID}}` and `{{.Prequel.LoadID}}` into the body and check them on receipt. Optionally set `webhook_record.max_records_per_minute` (integer, default `3000`) and `webhook_record.max_concurrency` (integer, default `1`) to control delivery throughput. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "users-record-destination", "type": "webhook_record", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" } }, "required": ["id", "email"] }, "webhook_record": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/records?provider={{.Prequel.ProviderID}}", "headers": { "Content-Type": "application/json" } } } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/records?provider=b6f2d1a4-3c58-4d0a-9e7b-2f1c8a9d4e63 Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro" } ``` Set `name` (string, required) to a unique identifier for this destination within your account. Set `type` (string, required) to `webhook_record` for record deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields each record contains. Prequel validates records against this schema before delivery. Set `webhook_record.request_template` (object, required) with `method`, `uri`, and optional `headers`. Use `{{.Prequel.ProviderID}}` in `uri` and `headers` to route by the provider whose data is being delivered. `.Record.*` fields and `.IsDeleted` are also available if you need to route by record content. See [Template variables](#template-variables) for the full list of available variables. Optionally set `webhook_record.max_records_per_minute` (integer, default `3000`) and `webhook_record.max_concurrency` (integer, default `1`) to control delivery throughput. #### Destination spec you register ```json title="POST /import/destinations" icon="brackets-curly" expandable theme={null} { "name": "acme-corp-record-destination", "type": "webhook_record", "record_schema": { "type": "object", "properties": { "id": { "type": "string" }, "email": { "type": "string" }, "subscription": { "type": "string" }, "tax_id": { "type": "string" } }, "required": ["id", "email", "tax_id"] }, "webhook_record": { "request_template": { "method": "POST", "uri": "https://example.com/prequel/records/acme-corp", "headers": { "Content-Type": "application/json" } } } } ``` #### What Prequel sends to your endpoint ```http title="Webhook request" expandable theme={null} POST https://example.com/prequel/records/acme-corp Content-Type: application/json X-Prequel-Webhook-Timestamp: 2026-01-01T00:00:00Z X-Prequel-Webhook-Signature: 4f81c9a2... X-Prequel-Webhook-Digest: sha256=9c1e5b8d... { "id": "usr_001", "email": "jane@acme.com", "subscription": "pro", "tax_id": "US-77-1234567" } ``` Create a separate destination for each provider whose requirements differ. Give each one a `name` (string, required) that identifies the provider it serves. Set `type` (string, required) to `webhook_record` for record deliveries. Set `record_schema` (object, required) to a [JSON Schema (draft-07)](https://json-schema.org/specification-links#draft-7) document that declares the fields this provider sends, including any custom fields unique to them. Prequel validates records against this schema before delivery. Set `webhook_record.request_template` (object, required) with `method`, `uri`, and optional `headers`. Because the destination is scoped to a single provider, you can hard-code provider-specific routing into `uri` and `headers` without needing template variables. Optionally set `webhook_record.max_records_per_minute` (integer, default `3000`) and `webhook_record.max_concurrency` (integer, default `1`) to control delivery throughput. ## Template variables The `body`, `uri`, and `headers` templates all have access to every variable below. | Variable | Type | Description | | --------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------- | | `{{.Record.}}` | `any` | Value of a specific field from the record (e.g. `{{.Record.email}}`). The field must be declared in `record_schema`. | | `{{.Record.AsJson}}` | `object` | The full record serialized as a JSON object. | | `{{.IsDeleted}}` | `boolean` | `true` when Prequel is delivering a record deletion, `false` otherwise. | | `{{.Prequel.ProviderID}}` | `string` | Identifier for the provider whose data is being delivered. | | `{{.Prequel.LoadID}}` | `string` | Identifier for the active load job. | | `{{.Prequel.DestinationID}}` | `string` | Identifier for the destination receiving this delivery. | | `{{.Prequel.StreamID}}` | `string` | Identifier for the stream this delivery came from. | | `{{.Prequel.RecordAcknowledgementURL}}` | `string` | URL to `PUT` this record's acknowledgement to. See [Acknowledgements](/import/features/acknowledgements). | | `{{.Prequel.TenantID}}` | `string` | Identifier for the tenant the stream belongs to. | | `{{.Prequel.IsTest}}` | `boolean` | `true` when the delivery comes from a test load. | ## Webhook headers Prequel includes the following headers on every delivery request. | Header | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `X-Prequel-Webhook-Signature` | SHA-256 RSA PKCS1 v1.5 signature of the payload. To verify, fetch the public key from `GET /public/signatures/webhook-public-key` and check the hex-decoded signature against the SHA-256 of `.` using PKCS1 v1.5. | | `X-Prequel-Webhook-Timestamp` | RFC 3339 timestamp of when the request was sent. Reject stale timestamps to prevent replay attacks. | | `X-Prequel-Webhook-Digest` | SHA-256 hash of the raw request body. | | `Content-Type` | For batch deliveries, defaults to `application/json` unless overridden by the `headers` template. Record deliveries send only the headers the `headers` template specifies. | | `X-Prequel-Is-Test` | Sent with value `true` only on deliveries from a test load. Acknowledgements uploaded for test deliveries are not ingested. | | `X-Prequel-Idempotency-Key` | Unique idempotency key across retry attempts of the same batch or record within a load job. | | `Content-Type` | Defaults to `application/json` unless overridden by the `headers` template. | ## Response codes Your endpoint should return the following response codes. | Code | When to return it | What Prequel does | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `2xx` | You accepted the record. | Records the delivery as successful. | | `429`, `503`, `504` with a `Retry-After` header | You are rate limited or temporarily unavailable and want the delivery attempted again. | Waits the interval the header specifies, capped at 30 seconds, then retries, for a maximum of three attempts per delivery. | | `400`, `412`, `413`, `422` | The payload itself is the problem. Return `422` for schema or data validation failures, `413` when it exceeds your size limit, and `400` when the request is malformed. | Records the delivery as rejected by your endpoint. Does not retry. | | Any other status | Your system could not accept the delivery. | Records the delivery as failed at the destination. This includes `500` and `502`, and a `429`, `503`, or `504` whose `Retry-After` header is absent or unparseable. Does not retry. | For more detail on this retry behavior see: Retry rules, redelivery across loads, and the circuit breakers that stop a failing stream. ## Error codes Prequel records the [error code](/import/logging/error-codes) below against the record, so return the status that best describes the failure. Use `422` for schema and data validation problems. | Status | Error code | Title | | ----------- | ----------- | ---------------------------------------- | | `400` | `PRQL-JPDQ` | Import Destination Bad Request | | `403` | `PRQL-CNJK` | Import Destination Forbidden | | `404` | `PRQL-VFTB` | Import Destination Not Found | | `413` | `PRQL-QAHC` | Import Destination Payload Too Large | | `422` | `PRQL-NROT` | Import Destination Unprocessable Entity | | `429` | `PRQL-WQKF` | Import Destination Rate Limited | | `431` | `PRQL-QFJW` | Import Destination Headers Too Large | | `500` | `PRQL-PHOK` | Import Destination Internal Server Error | | `502` | `PRQL-QPMF` | Import Destination Bad Gateway | | `503` | `PRQL-HNNX` | Import Destination Service Unavailable | | `504` | `PRQL-PRHQ` | Import Destination Gateway Timeout | | Other `4xx` | `PRQL-IMEK` | Import Destination Client Error | | Other `5xx` | `PRQL-TNXG` | Import Destination Server Error | ## Acknowledgements Without an acknowledgement, a record your pipeline rejects an hour later still counts as delivered. Acknowledging makes that failure visible to you and to the Provider who can fix it. Report each record's downstream outcome back to Prequel once your system has processed it. # React SDK Source: https://docs.prequel.co/import/developer-sdks/react-sdk Use our React SDK to quickly integrate Prequel Import into your React application # React SDK Use `@prequel/import-react` to build custom Import experiences in a React application. The SDK provides a provider, a default authenticated client, and hooks for sources, datasets, streams, extracts, loads, and destinations. ## Install ```sh theme={null} npm install @prequel/import-react react ``` ## Authentication Do not expose your Prequel API key to browser code. Instead, create a backend endpoint that exchanges your server-side credentials for a short-lived scoped Import API token, then pass that token fetcher to `createDefaultClient`. ```tsx theme={null} import { ImportProvider, createDefaultClient } from "@prequel/import-react"; const client = createDefaultClient({ fetchToken: async () => { const response = await fetch("/api/prequel/import-token"); const { token } = await response.json(); return token; }, }); export function App() { return ( ); } ``` `createDefaultClient` caches the token and retries once with a fresh token after a `401` response. ## Query Hooks List hooks return pagination helpers in addition to the current page of items. ```tsx theme={null} import { useListDatasets } from "@prequel/import-react"; export function DatasetList() { const { items, isLoading, error, hasNextPage, fetchNextPage } = useListDatasets({ pageSize: 25, }); if (isLoading) return null; if (error) return

Unable to load datasets.

; return ( <>
    {items.map((dataset) => (
  • {dataset.name}
    • ))}
); } ``` Use single-resource hooks when you already have an ID: ```tsx theme={null} import { useSource, useSourceTables } from "@prequel/import-react"; export function SourceDetails({ sourceId }: { sourceId: string }) { const source = useSource(sourceId); const tables = useSourceTables(source.data?.id); if (source.isLoading || tables.isLoading) return null; return 
{JSON.stringify({ source: source.data, tables: tables.tables }, null, 2)}
; } ``` ## Action Hooks Create, update, preview, and test hooks return action state plus a named function for the operation. ```tsx theme={null} import { useCreateSource, type SourceOptions } from "@prequel/import-react"; export function CreateSourceButton({ source }: { source: SourceOptions }) { const { createSource, isCreating, error } = useCreateSource(); return ( <> {error ?

{error.message}

: null} ); } ``` Each action hook also exposes `execute`, which returns `{ ok, data, error }` instead of throwing. ## Source Forms Use `useSourceForm` to build a vendor-specific source connection form from Prequel metadata. It returns the fields to render, a draft source object, a setter, and `getSource()` for producing a typed create/update payload. ```tsx theme={null} import { useSourceForm } from "@prequel/import-react"; export function SourceForm({ providerId, vendor }: { providerId: string; vendor: string }) { const form = useSourceForm({ providerId, vendor }); if (form.isLoading) return null; return (
{form.fields.map((field) => ( ))} ); } ``` ## Hooks | Workflow | Hooks | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Datasets | `useListDatasets`, `useDataset`, `useDatasetSchema`, `useRandomSampleDataset`, `useCreateDataset` | | Sources | `useListSources`, `useSource`, `useSourceTables`, `useSourceSchema`, `usePreviewSourceRows`, `useCreateSource`, `useUpdateSource`, `useTestNewSource`, `useTestExistingSource`, `useSourceForm`, `useSourceFormFields` | | Streams | `useListStreams`, `useStream`, `useCreateStream`, `useUpdateStream`, `useTestNewStream`, `useTestExistingStream`, `useTestExpression` | | Destinations | `useListDestinations`, `useDestination` | | Extracts | `useListExtracts`, `useExtract`, `useExtractDebug` | | Loads | `useListLoads`, `useLoad` | ## Types The SDK exports TypeScript types for request payloads and response resources, such as `SourceOptions`, `DatasetResource`, and `StreamResource`. Use these from your editor while building forms and handling responses. For signatures and return shapes, see the [React SDK Reference](/import/developer-sdks/react-sdk-reference/react-sdk) and [React SDK Common Types](/import/developer-sdks/react-sdk-reference/common-types). For the API model shapes used by the hooks, see [React SDK API Models](/import/developer-sdks/react-sdk-reference/api-models). For endpoint-level schemas, see the [Import API reference](/import/api-reference/introduction). # Acknowledgements Source: https://docs.prequel.co/import/features/acknowledgements Report each record's downstream outcome to Prequel after your system finishes processing it A successful delivery confirms only that your endpoint received a record. An acknowledgement reports what your system did with it afterward. Acknowledgements are for delivery tracking and observability, not control flow. An acknowledgement that reports failure does not trigger a retry or a redelivery. To understand retry and redelivery, see [Delivery failure handling](/import/features/delivery-failure-handling). ## When to use acknowledgements 1. **[Batch delivery](/import/destination-specs/webhook-batch):** A [status code](/import/destination-specs/webhook-batch#response-codes) covers the whole batch file, so acknowledgements are the only way Prequel learns the outcome of any single record inside it. 2. **Asynchronous ingestion:** When your endpoint queues a record and responds before processing it, the acknowledgement carries the real verdict back. * Return a `2xx` on receipt so the delivery is not retried, then acknowledge with a record-level status code once processed asynchronously. 3. **Provider feedback:** Downstream failures become available per record, which you can optionally surface to Providers so they correct the offending records at the source. See [Replication monitoring](/import/features/replication-monitoring). Acknowledgements are valuable for record delivery to handle asynchronous ingestion. With batch delivery, acknowledgements are required in order to provide record-level statuses to your users on their source data. ## Where to send them Every delivery carries an acknowledgement URL. Make an HTTP `PUT` to that URL with your acknowledgement as the body. Where the URL arrives, and the `Content-Type` to send, depend on the delivery type. | Delivery type | Where the URL arrives | `Content-Type` | | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | [Record](/import/destination-specs/webhook-record) | The [`{{.Prequel.RecordAcknowledgementURL}}`](/import/destination-specs/webhook-record#template-variables) template variable. Template it into your request so your endpoint receives it. | `application/json` | | [Batch](/import/destination-specs/webhook-batch) | The [`batch_acknowledgement_url`](/import/destination-specs/webhook-batch#delivery-format) field on the delivery envelope. | `application/jsonlines` | Send the `Content-Type` shown above or the request is rejected. Each URL expires 24 hours after the delivery. ## What to send A record delivery takes a single acknowledgement object. A batch delivery takes newline-delimited JSON, one acknowledgement per line. The example below is expanded for readability: ```json title="Example" icon="brackets-curly" expandable theme={null} { "version": "v0", "data": { "record_id": "usr_001", "batch_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "status_code": 422, "error_code": "MISSING_TAX_ID", "error_message": "tax_id is required for records in this region", "timestamp": "2026-01-01T00:00:00Z" } } ``` Schema version of the acknowledgement. Currently `v0`. Identifies the record being acknowledged. The HTTP status code your system reached for this record. Use `200`, `201`, or `204` to mark it succeeded, or a `4xx` or `5xx` to mark it failed. RFC 3339 timestamp of when your system reached this outcome. When a record has more than one acknowledgement, the latest `timestamp` takes precedence, so sending again re-acknowledges the record. The batch the record was delivered in, taken from [`batch_id`](/import/destination-specs/webhook-batch#delivery-format) on the delivery envelope. A description of the failure. Kept only when `status_code` is `400` or above. An acknowledgement that fails schema validation is not dropped; it marks the record as errored. Use the dataset's [`primary_key_column`](/import/core-concepts/datasets#1-dataset-type) value as the `record_id`. Batch deliveries carry it as [`prequel__record_id`](/import/destination-specs/webhook-batch#delivery-format) on every record in the batch file, except when a `body` template shapes a `json` batch. In that case the template controls the payload, so render the primary key into it with `{{.Record.}}` or you will have no id to acknowledge against. [Get record](/import/api-reference/import-datasets/get-import-dataset-record) returns a record by that id, including its acknowledgement state. ## When acknowledgements appear Prequel ingests acknowledgements automatically, typically within 15 minutes of the [load](/import/core-concepts/dataflow#lifecycle-of-a-transfer) completing, and only those sent within 75 minutes of it. Once ingested, a record's downstream outcome sits alongside its delivery history in [Get record](/import/api-reference/import-datasets/get-import-dataset-record) and [load debugging](/import/logging/dataflow-debugging), so you can trace one record from extraction through to your system's final verdict. Optionally, you can use the [initialize acknowledgement endpoint](/import/api-reference/import-streams/initialize-acknowledge) to ingest on demand. Use this when your processing finishes outside the 75-minute window. **The endpoint does not accept acknowledgements;** it tells Prequel to process acknowledgements you have already sent, optionally bounded by `start_time` and `end_time` on when you sent them. # Change detection Source: https://docs.prequel.co/import/features/change-detection Learn how Prequel Import detects and syncs changes from your data sources Prequel Import supports a variety of sources and makes minimal assumptions about the structure of the data in these sources to maximize compatibility. That said, there are some minimum requirements and optional enhancements that enable efficient change detection and syncing to destinations. ## Minimum requirements for source data ### Unique row identifier (for example, a primary key) Every source table must have some unique row identifier. This is needed to trace unique rows through the pipeline and to efficiently detect new and updated records. The Unique Row Identifier can be a "synthetic" column (such as a concatenation or a hash of some other columns). The only requirement is that it is unique and unchanging. ### Updated at timestamp (required for fact tables) For source tables under \~10 million total rows, Prequel can detect data changes without the use of an explicit Updated At timestamp. ## Optional enhancements ### Updated at timestamp (optional for dimension tables) For more efficient change detection or to support volumes greater than 10m total rows, an Updated At column can greatly improve efficiency. Fact tables require a Updated At column (semantically, this would be synonymous with Created At) as a best practice. If a fact table does not have an Updated At column, it can be treated as a dimension table, but at the cost of lower efficiency. ## Compatibility grid | Dataset Type | Unique Row Identifier | Updated At Column | Volume Limits | | ------------ | --------------------- | ----------------- | -------------------------------------------------------------------- | | Fact | Required | Required | 100m rows per day | | Dimension | Required | Optional | 100m rows per day with Updated At, 10m rows total without Updated At | Volume limits are based on current performance characteristics and may be revised in the future. # Delivery failure handling Source: https://docs.prequel.co/import/features/delivery-failure-handling How Prequel recovers a failed delivery without data loss or manual backfills Prequel has intentional retry, redelivery, and circuit breaking logic designed to protect data integrity in your pipeline and minimize the operational burden of failed deliveries. ## Retries Prequel retries delivery to your destination for only `429`, `503`, and `504` responses that carry a `Retry-After` header, for a maximum of three attempts per delivery. Every other response, and every network failure, is attempted once. For which status code to return and the error code Prequel records for it, see the [Record API](/import/destination-specs/webhook-record#response-codes) or [Batch API](/import/destination-specs/webhook-batch#response-codes) spec. The `Retry-After` value (in seconds) sets the wait before the next attempt, capped at 30 seconds. ## Redelivery Each load selects every record that has not been delivered yet, plus any record whose content changed since its last delivery. A record that failed to reach your endpoint therefore remains eligible and is delivered again on the next scheduled load, with no backfill required. Because a record can be delivered more than once across attempts and loads, your endpoint must be idempotent. In your pipeline, deduplicate on the dataset's [`primary_key_column`](/import/core-concepts/datasets#1-dataset-type). Every delivery also carries an [`X-Prequel-Idempotency-Key`](/import/destination-specs/webhook-record#webhook-headers) header, which stays the same across retry attempts of the same request. ### Rejected records A `5xx` means your endpoint could not accept the delivery. A `4xx` means something is wrong with the data being delivered, so Prequel treats a `400`, `412`, `413`, or `422` as a rejection. Prequel builds the batch request itself, so on a batch delivery every error status indicates the records were not delivered rather than that any record was bad. For these records, what happens on the next load depends on the delivery type: | Delivery type | Delivery status | Behavior on next load | | -------------------------------------------------- | -------------------------------------------- | ----------------------------------------------------------- | | [Record](/import/destination-specs/webhook-record) | Delivered, with the rejection's error code | Not delivered again until its content changes at the source | | [Batch](/import/destination-specs/webhook-batch) | Not delivered, for every record in the batch | Every record in the batch is delivered again | A record counts as delivered once your endpoint receives it, whether or not your endpoint accepted it. The outcome of an individual record in a batch reaches Prequel only through [acknowledgements](/import/features/acknowledgements). ## Circuit breaking Retries and redelivery assume a failure will clear on its own. When it will not, such as a broken mapping or an unreachable endpoint, circuit breakers stop the pipeline and report the cause to preserve resources on your [Provider's](/import/core-concepts/providers) source and on your [destination](/import/core-concepts/destinations). **Subscribe a [webhook](/import/logging/webhooks-and-monitoring) to be alerted when an extract or load fails.** ### Source circuit breaker Prequel halts an [extract](/import/core-concepts/dataflow#lifecycle-of-a-transfer) from the [source](/import/core-concepts/sources), or a load to the destination, when 100 or more of its records fail, which indicates a mapping or schema mismatch at the source. The next scheduled extract or load runs as usual, and trips again if the same records still fail. ### Destination circuit breaker Prequel sets `is_circuit_broken` on the [stream](/import/core-concepts/streams) when a load fails with a destination-side error, such as an unreachable host, a `403` or `404`, or a `5xx` response. The failure ends the load even when other records in it were delivered successfully, and cancels the delivery work still queued behind it. A load sends one request per record for [record delivery](/import/destination-specs/webhook-record), or one per batch for [batch delivery](/import/destination-specs/webhook-batch), split by [`max_size_per_batch`](/import/destination-specs/webhook-batch#creating-your-destination-spec). Every record in a batch shares that batch's outcome, so a halt leaves a load's records in three states: | Records within the halted load | Delivery status | Behavior on next load | | ---------------------------------------- | --------------- | --------------------- | | Accepted with a `2xx` before the failure | Delivered | Not delivered again | | In the request that ended the load | None recorded | Delivered again | | Never sent, because the load halted | None recorded | Delivered again | While the flag is set, extracts continue to run but produce no loads, and manual load triggers are rejected. To resume loads, set it back to `false` with [Update import stream](/import/api-reference/import-streams/update-import-stream). # Replication monitoring Source: https://docs.prequel.co/import/features/replication-monitoring Track sync status, monitor data freshness, and troubleshoot replication issues The Prequel Import Destination specs specify detailed error codes on a per row/record level. This enables granular debugging at both the individual transfer level, but also one level deeper at the row level. ## Monitoring in the Admin UI All sync issues are raised in the Prequel Import Admin UI, and can be drilled down into the individual rows. Because Prequel Import does not persist pipeline data in the application database, some granular queries may take longer to complete while details are retrieved from the Datalake cache. ## Monitoring synchronously For live error detection, Prequel Import emits webhooks for most application events. These webhooks can be delivered to a variety of monitoring platforms including Slack, PagerDuty, Datadog, and more. See the [Webhooks and Monitoring](/import/logging/webhooks-and-monitoring) guide for details. ## Cancelling in-flight jobs Running extracts and loads can be cancelled via the API: | Method | Path | Purpose | | ------ | ------------------------------ | --------------------------------------------------------------- | | `POST` | `/import/extracts/{id}/cancel` | Cancel a running extract. Returns the updated extract resource. | | `POST` | `/import/loads/{id}/cancel` | Cancel a running load. Returns the updated load resource. | Cancellation is best-effort. A job that has already reached a terminal state will be returned unchanged. ## Provider-facing monitoring All monitoring data is designed to be securely displayed to Providers in embedded experiences. See the [Customer Experience](/import/getting-started/customer-experience) guide for details. # Advanced table mapping Source: https://docs.prequel.co/import/features/table-mapping Configure custom mappings between Source Datasets and target Destination schemas Prequel Import anticipates the case where Source Dataset schemas do not perfectly match target Destination schemas without some mapping and transformation. To support bridging between the two, Prequel supports a number of mapping and transformation options. ## Who's responsible for performing the mapping? Prequel Import provides tooling to support both the Destination owner (you) to perform the mapping, as well as to expose this experience to your users. If you plan to embed this experience for your customers, please see the [Customer Experience](/import/getting-started/customer-experience) guide. Dataset Mapping ## Custom queries At a full table level, custom SQL queries can be used to transform the Source data. This is a good solution for users that are comfortable with SQL, or who need aggregations or logic that spans multiple rows. ## Mapping categories Each Stream mapping has a `field_type` that determines how the target Destination field is populated. The two categories are summarized below. A `field` mapping copies a Source column value directly into a target Destination field. An `expression` mapping computes the target value from one or more Source columns using a formula (expression). See the example below for a Stream that combines column and transformation mapping. ```bash title="POST /streams" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/streams \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "stream": { "name": "users-stream", "dataset_id": "", "destination_id": "", "mappings": [ { "target_field": "id", "field_type": "field", "column": { "name": "id" } }, { "target_field": "email", "field_type": "field", "column": { "name": "email_address" } }, { "target_field": "customer_segment", "field_type": "expression", "expression": { "expr": "IF(annual_revenue >= 100000, \"enterprise\", IF(annual_revenue >= 10000, \"mid_market\", \"smb\"))" } }, { "target_field": "full_name", "field_type": "expression", "expression": { "expr": "CONCATENATE(first_name, \" \", last_name)" } }, { "target_field": "signup_date", "field_type": "expression", "expression": { "expr": "TIMESTAMPVALUE(created_at)" } }, { "target_field": "source_system", "field_type": "expression", "expression": { "expr": "\"crm\"" } } ] } }' ``` ## Expression syntax Expressions use the [expr-lang](https://expr-lang.org/docs/language-definition) language. Any operator or built-in function from the expr-lang language definition is supported. ## Additional supported functions Prequel Import also provides the following additional helper functions for common transformations: | Function | Signature | Description | | ---------------- | ------------------------------------------------ | ----------------------------------------------------------------------- | | `TIMESTAMPVALUE` | `TIMESTAMPVALUE(text)` | Parses a date or timestamp string. | | `IF` | `IF(condition, true_val, false_val) -> any` | Returns one of two values based on a condition. | | `TODAY` | `TODAY() -> string` | Returns today's date. | | `TEXTSPLIT` | `TEXTSPLIT(text, delimiter) -> []string` | Splits text on a delimiter. | | `PROPER` | `PROPER(text) -> string` | Capitalizes the first letter of each word. | | `SUBSTITUTE` | `SUBSTITUTE(text, old_text, new_text) -> string` | Replaces occurrences of `old_text` with `new_text`. | | `LEFT` | `LEFT(text, num_chars) -> string` | Returns the leftmost characters of a string. | | `RIGHT` | `RIGHT(text, num_chars) -> string` | Returns the rightmost characters of a string. | | `MID` | `MID(text, start_num, num_chars) -> string` | Returns a substring starting at a given position. | | `LEN` | `LEN(text) -> int` | Returns the length of a string. | | `FIND` | `FIND(find_text, within_text) -> int` | Returns the position of `find_text` in `within_text`. Case-sensitive. | | `SEARCH` | `SEARCH(find_text, within_text) -> int` | Returns the position of `find_text` in `within_text`. Case-insensitive. | | `TEXT` | `TEXT(value, format_text) -> string` | Formats a value as a string using the given format. | | `VALUE` | `VALUE(text) -> float64` | Parses a numeric string into a float. | | `CONCAT` | `CONCAT([str]...) -> string` | Concatenates the given strings. | | `CONCATENATE` | `CONCATENATE([str]...) -> string` | Concatenates the given strings. | | `SHA256` | `SHA256(text) -> string` | Returns the SHA-256 hash of the input string. | | `AND` | `AND([expr]...) -> bool` | Returns `true` if all arguments evaluate to `true`. | | `OR` | `OR([expr]...) -> bool` | Returns `true` if any argument evaluates to `true`. | ## Object storage virtual columns For object storage and SFTP sources, every extracted record includes two virtual columns derived from the object's storage metadata, such as the object key and its last modified time. Reference them in mappings and expressions by name, like any other Source column. For example, an expression can extract a report date from the object key when the rows themselves do not contain one. See [Datasets](/import/core-concepts/datasets#2-dataset-method) for how these columns are populated and their role in change detection. | Column | Prequel Type | Description | Example | | -------------------------- | ------------ | --------------------------------------- | ------------------------------ | | `meta__filename` | `string` | Key or path of the source object | `"exports/2023/users.parquet"` | | `meta__file_last_modified` | `timestamp` | Last modified time of the source object | `"2023-01-01T14:30:00Z"` | # Ultra-High volumes Source: https://docs.prequel.co/import/features/ultra-high-volumes Scale your data imports to handle enterprise-level throughput Prequel Import is designed to support extremely high volumes of data. To support these high volumes, a few advanced configuration options may be useful, or in some cases, necessary. ## Updated at column For volumes exceeding \~10m rows, an explicit [Updated At column](/import/features/change-detection#optional-enhancements) that records the time that the row was last modified significantly improves change detection performance. ## Batch API destination type If the target Destination endpoint is expected to receive more than \~10m rows per day for any Provider, we strongly recommend using the [Batch API Destination](/import/core-concepts/destinations) type for increased efficiency. ## Resource tuning By default, your Prequel Import account is provisioned with a default hardware selection for average transfer workloads. However, there are cases where the underlying infrastructure must be tuned to properly support workloads that are significantly larger than average. Please contact your Prequel Representative if you suspect this may be applicable to your use case. # Building your customer experience Source: https://docs.prequel.co/import/getting-started/customer-experience Create a connected data import experience for your customers # Recommended UX There are two primary user experiences to consider when designing your embedded Data Import feature: the ***Setup Connection*** experience and ***Monitoring*** experience. ## Setup connection experience Prequel Import exposes the APIs and SDKs you need to create your own setup flow. Our SDK is designed to support a Setup Connection experience that resembles the experience in the Admin UI, and consists of 3 parts: Guide the user to select one of the supported Sources (e.g., Databricks, Snowflake, S3, etc.). ![Source Selection](https://storage.googleapis.com/prequel_docs/images/import-source-selection.png) Collect connection details and provide connection test tools. ![Connection Details](https://storage.googleapis.com/prequel_docs/images/import-connection-details.png) Guide the user to map their selected Source to a supported Destination, and support the mapping of columns to fields accordingly. ![Dataset Mapping](https://storage.googleapis.com/prequel_docs/images/import-dataset-mapping-part1.png) ![Dataset Mapping](https://storage.googleapis.com/prequel_docs/images/import-dataset-mapping-part2.png) ## Monitoring experience Similarly, Prequel Import exposes the APIs and SDKs you need to create your own sync monitoring experience. For more details and design inspiration, see the [React SDK](/import/developer-sdks/react-sdk) guide. # Native embedding Prequel Import is designed to be fully embedded, and our API and developer SDKs make that simple. ### Developer APIs Use the Prequel Import API to build a fully custom connection experience for your customers. [View API documentation](/import/api-reference/introduction) ### Developer SDKs Use our developer SDKs to quickly integrate Prequel Import into your application. [View Developer SDKs](/import/developer-sdks/react-sdk) ## Alternatives to full embedding Prequel Import also supports alternative options for connecting and configuring Providers and their Sources. Use the Prequel Admin UI to manage Providers and Sources without writing code. Share secure, one-time links with your customers to let them configure their own connections. # How Prequel Import works Source: https://docs.prequel.co/import/getting-started/how-prequel-import-works Learn the core concepts behind Prequel Import and how it fits into your product Prequel Import helps you reliably sync your customer's data from their preferred data platform into your application. Prequel can help you power this as an embedded feature, reliably and securely across data platforms, arbitrary data models, and at extremely high volumes. ![Prequel Import architecture](https://storage.googleapis.com/prequel_docs/images/import-architecture.png) ## Requirements To use Prequel Import, you will need to provide an object storage bucket (S3, GCS, or Azure) and implement a spec-compliant API endpoint to receive delivered records. Kafka topic delivery is coming soon. Receive batches of records via a spec-compliant API endpoint. Recommended for most use cases. [Learn more about the Batch API](/import/destination-specs/webhook-batch) Receive individual records as they change via a spec-compliant API endpoint. [Learn more about the Record API](/import/destination-specs/webhook-record) Configure a Kafka topic to receive data from Prequel Import. [Learn more about the Kafka Topic spec](/import/destination-specs/kafka-topic) ## Configuring Prequel import Prequel Import is configured by defining the possible datasets or endpoints that you want to allow your users to push data into. Once configured, you can begin connecting to customers or data "providers" and start syncing data. ## Understanding Prequel import architecture Prequel Import works by regularly detecting changes in the provider's source, and reliably delivering those changes to the designated endpoint. Architecturally, this works by connecting to the source, maintaining a secure cache of the state of the source in object storage, and resiliently pushing detected changes to the designated endpoints. To learn more, also see the [Dataflow](/import/core-concepts/dataflow) guide. ## Exposing Prequel import to your users Prequel Import is optimized for a fully embedded experience, built natively into your platform. To support that implementation method, Prequel Import is API first, with a variety of developer tools. Besides the fully embedded UX, onboarding data providers can also be accomplished using our Admin UI. Magic Link onboarding is coming soon. To learn more, see the [Customer Experience](/import/getting-started/customer-experience) guide. # Quickstart guide Source: https://docs.prequel.co/import/getting-started/quickstart Configure a Prequel Import pipeline end to end and sync your first records This guide walks through the order in which to create resources for a complete Prequel Import pipeline. Each step links to the corresponding [API reference](/import/import-api/overview) page and includes an expandable curl example. ## Select your onboarding experience Get started with the Prequel agent skill, the Prequel MCP server, a Postman collection, or curl commands. Skills are an open standard for packaging reusable agent capabilities. Claude Code, Cursor, and other skill-aware agents can all load the Prequel Import skill. Open the [raw skill file](https://docs.prequel.co/.well-known/agent-skills/prequel-import-quickstart/SKILL.md), copy the contents, and save the file to your agent's skills directory. For Claude Code use `~/.claude/skills/prequel-import-quickstart/SKILL.md`. For Cursor use `.cursor/skills/prequel-import-quickstart/SKILL.md` in your project root and reload the workspace. Set up a new Prequel Import pipeline from scratch using the prequel-import-quickstart skill. Walk me through creating each resource in order, run the curl commands as we go, and finish by triggering an Extract and a Load to verify the pipeline end to end. Mintlify auto-hosts an MCP server for this documentation site, exposing search and filesystem tools your agent can call to look up the right operations and schemas. Add `https://docs.prequel.co/mcp` to your tool's MCP configuration. Most clients accept the URL directly through their MCP settings. Set up a new Prequel Import pipeline from scratch. Use the Prequel Import MCP server to look up the correct API operations and request schemas as we go, walk me through creating each resource in order, and finish by triggering an Extract and a Load to verify the pipeline end to end. In Postman, click **Import**, paste the [Prequel Import OpenAPI spec URL](https://docs.prequel.co/generated/openapi-import-generated-2023-12-01.json), and confirm. Postman generates a complete collection from the spec with one request per operation. Add a `PREQUEL_API_KEY` variable to the imported collection and reference it from each request's `X-API-Key` header. Browse the [API reference](/import/api-reference/introduction) and select the operation for the resource you want to create. Use the **Copy** button on each operation's example request body, paste it into your shell as the `-d` payload, and substitute your variables and resource IDs as you go. ## Prerequisites You will need a Prequel API key. If you do not have one, contact your Prequel representative to get access. The API key used in the `X-API-Key` header will route your request to either production or staging based on the key. ```bash title="Set your API key" icon="terminal" theme={null} export PREQUEL_API_KEY="" ``` Stand up a [spec-compliant API endpoint](/import/destination-specs/webhook-batch) to receive delivered records. Confirm your key works against the API. A 200 response means your environment is ready. ```bash title="Verify API access" icon="terminal" theme={null} curl -H "X-API-Key: $PREQUEL_API_KEY" https://api.prequel.co/ ``` ## Setup overview Prequel Import should follow a three phase setup to prevent errors from missing dependencies: 1. **Prepare your environment** by creating a Datalake. The Datalake is a one-time setup shared across all Providers. 2. **Configure and validate a data source** by creating a Provider, Source, and Dataset, then triggering an Extract to confirm the Source, Dataset, and Datalake are configured correctly before adding delivery. 3. **Connect and deliver** by standing up a delivery endpoint, creating a Destination (and optionally a custom Product), creating a Stream, testing the transform, and triggering a Load. ## Set up your pipeline Each call returns a resource ID in the response body. Record the IDs as you go, since later steps reference them with placeholders such as `` and ``. ### Phase 1: prepare your environment Provision a bucket on S3, GCS, Azure Blob Storage, or an S3-compatible store to back the Datalake, then create the Datalake referencing that bucket. Typically, a single multi-tenant Datalake is created and reused across Providers, since Prequel automatically partitions the bucket internally by Provider. You may create and assign additional Datalakes if you prefer bucket-level isolation for Providers. See [Datalake](/import/core-concepts/datalake) for configuration steps and open the [API reference](/import/api-reference/datalakes/create-datalake) for the full request schema. ```bash title="Create datalake" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/datalakes \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "datalake": { "name": "primary-datalake", "storage": "s3", "s3": { "bucket_name": "", "bucket_region": "us-east-1", "prefix": "prequel-import", "auth_method": "aws_iam_role", "aws_iam_role": { "aws_iam_role_arn": "" } } } }' ``` ### Phase 2: configure and validate a data source Repeat this phase for each Provider you onboard. Each Provider can have multiple Sources and multiple Datasets. A Provider represents a customer or tenant of your platform. The Provider references the Datalake by ID and the assigned Products. Pass the special `["all"]` Product to grant access to every Destination when customers share all Destinations. If a customer should only have access to a subset of Destinations, use Products to restrict Destinations by Provider. See [Providers](/import/core-concepts/providers), or open the [API reference](/import/api-reference/providers/create-provider) for the full request schema. ```bash title="Create provider" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/providers \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "provider": { "name": "Acme Corp", "id": "acme", "datalake_id": "", "products": ["all"] } }' ``` A Source is the Provider's database Prequel will read from. See [Sources](/import/core-concepts/sources) for the supported vendors and per-vendor fields and open the [API reference](/import/api-reference/sources/create-import-source) for the full request schema. ```bash title="Create source" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/sources \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": { "name": "acme-postgres", "vendor": "postgres", "provider_id": "", "postgres": { "host": "db.acme.example.com", "port": 5432, "database": "production", "auth_method": "password_auth", "password_auth": { "username": "prequel", "password": "" } } } }' ``` A Dataset represents a structured table of data that Prequel will replicate from the Source. The Dataset declares its extraction method (`table` for a single table, `sql` for a custom query, `glob` for object-storage paths) and certain special columns and configuration options. This step is repeated for each Provider's table to be imported. See [Datasets](/import/core-concepts/datasets) and [Change Detection](/import/features/change-detection) for incremental sync configuration, or open the [API reference](/import/api-reference/import-datasets/create-import-dataset) for the full request schema. ```bash title="Create dataset" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/datasets \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "dataset": { "source_id": "", "method": "table", "table": { "source_table_namespace": "public", "source_table_name": "users" }, "type": "dimension", "dimension": { "primary_key_column": "id", "last_modified_column": "updated_at" }, "frequency_minutes": 60 } }' ``` The Extract pulls Dataset rows into the Datalake. Trigger one now to confirm the Source, Dataset, and Datalake are configured correctly before moving on to delivery. A successful Extract verifies authentication to the Source, read access to the Dataset's table, and write access to the Datalake. Open the [API reference](/import/api-reference/datasets/initialize-extract) for the full request schema. ```bash title="Initialize extract" icon="terminal" theme={null} curl -X POST https://api.prequel.co/import/datasets//extract \ -H "X-API-Key: $PREQUEL_API_KEY" ``` ### Phase 3: connect and deliver Repeat for each Dataset/Destination pair. A Destination is one of your delivery endpoints. Each Destination is configured with a `record_schema` that describes the records your endpoint expects and a `request_template` that defines the URL and auth headers. See [Destinations](/import/core-concepts/destinations) and the [Batch API specification](/import/destination-specs/webhook-batch), or open the [API reference](/import/api-reference/destinations/create-import-destination) for the full request schema. Repeat this step for each Destination you need. Confirm your delivery endpoint from the prerequisites is reachable. Prequel validates connectivity during creation. ```bash title="Create destination" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/destinations \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "destination": { "name": "primary-destination", "type": "webhook_record", "record_schema": { "type": "object", "required": ["id"], "properties": { "id": { "type": "string" } } }, "webhook_record": { "request_template": { "uri": "https://your-app.example.com/prequel/records", "method": "POST" } } } }' ``` By default, every Destination you create is included in a built-in `all` Product. Skip this step and reference `all` when you create a Provider, unless you want to group Destinations differently. To define a custom group, create a Product and include the Destination IDs you want it to cover. Open the [API reference](/import/api-reference/products/create-import-product) for the full request schema. ```bash title="Create product" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/products \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "product": { "name": "primary-product", "destinations": [""] } }' ``` A Stream maps one Dataset to one Destination through field-level mappings. Before authoring the mappings, fetch the Destination's `record_schema` so you know which fields your mappings need to cover. See [Streams](/import/core-concepts/streams) and [Advanced Table Mapping](/import/features/table-mapping), or open the [API reference](/import/api-reference/import-streams/create-import-stream) for the full request schema. ```bash title="Create stream" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/streams \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "stream": { "name": "users-stream", "dataset_id": "", "destination_id": "", "mappings": [ { "target_field": "id", "field_type": "field", "column": { "name": "id" } } ] } }' ``` Apply the Stream's mappings to a sample of extracted records and confirm the output covers every required field in the Destination's `record_schema` with the correct type. Open the [API reference](/import/api-reference/streams/create-import-stream-test) for the full request schema. ```bash title="Test transform" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/streams//test-transform \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "record_ids": [""] }' ``` The Load applies the Stream's mappings and delivers records to the Destination. Open the [API reference](/import/api-reference/streams/initialize-load) for the full request schema. Then, verify records have reached your delivery endpoint. ```bash title="Initialize load" icon="terminal" expandable theme={null} curl -X POST https://api.prequel.co/import/streams//load \ -H "X-API-Key: $PREQUEL_API_KEY" \ -H "Content-Type: application/json" \ -d '{}' ``` **That's it.** Your Prequel Import pipeline is now configured to automatically extract data from the Source and load to the Destination. To onboard each additional Provider, repeat **Phase 2: Configure and validate a data source** and **Phase 3: Connect and deliver**. ## Next steps Wrap the API in an embedded setup and monitoring UX for your customers. Get notified on transfer failures and track sync health. Tune incremental sync behavior for `dimension` Datasets. Use expressions and complex mappings to reshape source data. # Import API Source: https://docs.prequel.co/import/import-api/overview Programmatic access to the Prequel Import platform. The Prequel Import API is under active development and subject to change. Contact your Prequel Representative before beginning implementation and with any questions. The Import API gives you programmatic control over the Prequel Import platform. # Overview Source: https://docs.prequel.co/import/index Documentation for Prequel Import - sync data from your customers' sources into your product This guide is written exclusively for **Prequel Import**. To learn more about **Prequel Export**, please refer to our [Prequel Export Guide](/export). Learn the core concepts behind Prequel Import and how it fits into your product. Configure a Prequel Import pipeline end to end and sync your first records. Create a connected data import experience for your customers. ## Advanced functionality Learn how Prequel Import detects and syncs changes from your data sources. Configure custom mappings between source tables and your destination schema. Track sync status, monitor data freshness, and troubleshoot replication issues. Scale your data imports to handle enterprise-level throughput. # Audit logs Source: https://docs.prequel.co/import/logging/audit-logs Track and review audit logs for your Prequel Import activity All activity on the Prequel Import platform is recorded in an immutable audit log. You can fetch audit log entries via the `/logs` API or via Reporting Export. ## Audit log data model | Field Name | Data Type | Description | | ----------------- | ----------- | ------------------------------------------------------------------------------------------------------------------ | | `id` | `string` | Unique audit log ID. | | `created_at` | `timestamp` | Timestamp when the event occurred, reported in UTC (RFC 3339). | | `event_type` | `string` | The type of event. See [Event types](#event-types) below. | | `resource_id` | `string` | The ID of the resource involved in the event. | | `actor` | `object` | The agent that triggered the event. Contains `actor_id` and `type` (`unknown`, `user`, `api_key`, or `provider`). | | `object` | `json` | Nullable JSON object representing the current state of the resource. `DELETE` events will not populate this field. | | `previous_object` | `json` | Nullable JSON object representing the prior state of the resource. `CREATE` events will not populate this field. | ## Fetching logs Use `GET /logs` to list audit log entries, or `GET /logs/{log_id}` to retrieve a single entry by ID. ### Filtering and pagination `GET /logs` accepts the following query parameters: | Parameter | Description | | ------------ | --------------------------------------------------------------------- | | `event_type` | Restrict results to a single event type. | | `min_time` | Unix timestamp; return only events at or after this time. | | `count` | Number of logs to return. | | `offset` | Offset of the result window. | | `page_size` | Items per page. | | `order` | `asc` or `desc`. | | `cursor` | Cursor returned by a prior page; pass through to fetch the next page. | Successful responses include `has_next` and `next_url` alongside the `data` envelope to drive cursor-based pagination. ## Event types The `event_type` field is one of: * Datalakes: `IMPORT_DATALAKE_ADD`, `IMPORT_DATALAKE_UPDATE`, `IMPORT_DATALAKE_DELETE` * Providers: `IMPORT_PROVIDER_ADD`, `IMPORT_PROVIDER_UPDATE`, `IMPORT_PROVIDER_DELETE` * Sources: `IMPORT_SOURCE_ADD`, `IMPORT_SOURCE_UPDATE`, `IMPORT_SOURCE_DELETE` * Datasets: `IMPORT_DATASET_ADD`, `IMPORT_DATASET_UPDATE`, `IMPORT_DATASET_DELETE` * Destinations: `IMPORT_DESTINATION_ADD`, `IMPORT_DESTINATION_UPDATE`, `IMPORT_DESTINATION_DELETE` * Products: `IMPORT_PRODUCT_ADD`, `IMPORT_PRODUCT_UPDATE`, `IMPORT_PRODUCT_DELETE` * Streams: `IMPORT_STREAM_ADD`, `IMPORT_STREAM_UPDATE`, `IMPORT_STREAM_DELETE` # Dataflow debugging Source: https://docs.prequel.co/import/logging/dataflow-debugging Debug and troubleshoot your Prequel Import data pipelines Monitoring and maintaining connected sources is an important part of the Prequel Import product. There are 3 different stakeholders that may need to be involved, and different recommended tooling for each. ## Admin monitoring & debugging By default, we expect Prequel Admins to use the Admin UI to monitor and investigate issues. The Admin UI provides deep, record level details of sync issues to investigate issues. For live (push) notifications, Prequel emits Webhooks and offers first-class integrations with services such as Slack, PagerDuty, Datadog, and others. ## Provider monitoring & debugging Prequel Admins may not always be the right stakeholder for monitoring and acting on connection issues. For this reason, Prequel exposes APIs that enable the same Admin UI granularity in a Provider-facing experience (scoped to Provider-specific resources). # Error codes Source: https://docs.prequel.co/import/logging/error-codes Look up what a Prequel error code means when an import fails Every Prequel error maps to a stable code in the `PRQL-XXXX` format. Codes are shared across Prequel Import and Prequel Export, so the table below is the same reference you will find in the Export docs, and it includes codes that only apply to exports. ## Where codes appear A failed extract or load carries a full `error` object. ```json title="Error object on an extract or load" icon="brackets-curly" expandable theme={null} { "error": { "error_code": "PRQL-KFSS", "title": "Source Circuit Breaker Tripped", "blame": "first_party", "documentation_url": "https://docs.prequel.co/export/error-handling/error-codes", "message": "The job encountered too many source mapping, transform, or validation errors and triggered the circuit breaker. Execution was halted early to protect system resources. Please review the failed records to determine the root cause.", "trace": "..." } } ``` Individual records report a narrower shape. Each stage of a record's lifecycle reports its own `error_code` and `error_trace`, so you can tell whether a record failed during extract, transform, validate, acknowledge, or load. ```json title="Error on a record stage" icon="brackets-curly" expandable theme={null} { "error": { "error_code": "PRQL-RSLM", "error_trace": "..." } } ``` For a field-by-field breakdown of the error object, including what each `blame` value means, see [Coded error handling](/export/error-handling/error-handling) in the Export docs. ## Code reference | Code | Title | Message | | :-------- | :---------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PRQL-RZBD | All Records Errored | None of the source provided rows were processed successfully. Please review the error records to determine the cause of the failure. | | PRQL-BNOD | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please check the s3:ListBucket permission. | | PRQL-DDAK | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please check for Create Database on Catalog permission. | | PRQL-FDEW | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. | | PRQL-RWCV | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please verify the glue:DeleteTable permission exists on the generated tables. | | PRQL-VDMC | Athena Authorization Failed | The Athena engine encountered an error. Please check that the provided connection principal has been granted sufficient permissions. Specifically, please check the s3:GetObject permission. | | PRQL-XZWE | Athena Cannot Find Requested Iceberg Entity | Athena has detected a corrupted Iceberg table. Full refresh to resolve. | | PRQL-ONNN | Athena Connection Refused | The Athena connection was refused. Please check that the provided Athena access details are correct. | | PRQL-QDPO | Athena DDL Execution Timeout Error | The Athena query exceeded the engine's DDL query timeout. If this error persists, please contact support. | | PRQL-XJWM | Athena Data Catalog Timeout Error | The Athena data catalog connection timed out. If this error persists, please contact support. | | PRQL-AFOK | Athena Data Processing Failed | The Athena engine encountered a ConcurrentModificationException while executing a query. | | PRQL-AWRI | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process Iceberg formatted data. | | PRQL-BKDL | Athena Data Processing Failed | The Athena query engine encountered an error. A value found in the underlying source data is incompatible with the type double defined in the Athena table schema. Please review the source's data type and either cast the type or update your Prequel data model accordingly. | | PRQL-BSAX | Athena Data Processing Failed | The Athena engine encountered an error related to the timestamp precision on some timestamp data. | | PRQL-CFGX | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process data in hive mode. | | PRQL-FLUU | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process Iceberg formatted data. | | PRQL-GVZM | Athena Data Processing Failed | The Athena engine encountered a problem while attempting to process Hive formatted data. | | PRQL-CERZ | Athena Federated Read Error | Athena encountered an exception reading from a federated data source. Please refer to Athena logs for more information. | | PRQL-KXAL | Athena Glue Data Catalog Rate Limit Exceeded | AWS throttled Glue Data Catalog API requests for this Athena connection. This is usually transient; please retry, and reduce concurrent Athena operations if it persists. | | PRQL-DJDS | Athena Glue Metastore Operation Timeout | The Glue Data Catalog timed out fetching table metadata for the Athena source. Please review Glue API usage from concurrent operations, reduce contention if possible, and if necessary, request an increase in service quota. | | PRQL-JNZR | Athena IAM Role Token Audience Mismatch | AWS rejected the federated identity token because its audience did not match what the IAM role expects. Please verify the oaud condition in the role's trust policy matches the provided value. | | PRQL-YOTO | Athena Iceberg Partitions Exceeded | The Athena engine exceeded the maximum number of open partition writers. Please contact support for more information. | | PRQL-IVCA | Athena Invalid Cast Argument | One or more column was unable to be cast to the data type from the corresponding model. Please ensure that are source column data types are compatible with model data types and try again. | | PRQL-MSQX | Athena Invalid Date Value | Athena could not cast a data value to type DATE. Check the source data to ensure dates are in the format YYYY-MM-DD. | | PRQL-TKVM | Athena Invalid Timestamp Value | Athena could not cast a data value to type TIMESTAMP. Check the source data to ensure the data can be properly cast into timestamp or determine if it requires additional handling. | | PRQL-LFDR | Athena Lake Formation Permissions Error | The connection encountered one or more Lake Formation permissions errors. Please ensure any Lake Formation policy contains sufficient permissions and try again. Contact support if the issue persists. | | PRQL-LRCT | Athena Lambda Timeout | Athena encountered an exception reading from a federated data source. A lambda function timed out. | | PRQL-CKSC | Athena Missing Column | The Athena query engine encountered an error. A column name was not recognized. | | PRQL-KZWP | Athena NOT NULL Value Constraint Failed | A NULL value was found in a column with non-null requirements. Please check the model's columns designated as is\_primary\_key and is\_last\_modified for any null values in your source. | | PRQL-XIEA | Athena Query Timeout | The Athena query execution timed out. If this error persists, please contact support. | | PRQL-NXQP | Athena Read Error | The Athena engine encountered a problem while attempting to read the source view. Please confirm the view is not stale. | | PRQL-FJOD | Athena Runtime Encountered Error | The provided AWS role does not have access to the Athena Workgroup. If the Athena Workgroup does not exist, the provided role does not have permission to create the workgroup. | | PRQL-TJCZ | Athena Runtime Encountered Error | The Athena connection encountered a memory limit. | | PRQL-VKEV | Athena Runtime Encountered Error | The Athena catalog encountered an internal server error. If this error persists, please contact support. | | PRQL-WJTE | Athena Runtime Encountered Error | The Athena connection encountered an error. | | PRQL-WRFA | Athena Runtime Encountered Error | The Athena engine encountered an error. | | PRQL-YTNS | Athena Runtime Encountered Error | The Athena connection encountered a resource limit. | | PRQL-ASFB | Athena S3 Access Denied | The provided credentials were denied access to the S3 source bucket. Please check that the IAM role has s3:GetObject, s3:ListBucket, and s3:GetBucketLocation granted on the source bucket and its contents. | | PRQL-CMYT | Athena Schema Does Not Exist Error | The Athena schema could not be found. Please check that the schema exists and that the provided connection principal has been granted sufficient permissions. | | PRQL-HDXP | Athena Server Encountered Generic Error | The Athena connection encountered a generic internal error. If this error persists, please contact support. | | PRQL-KLBO | Athena Server Encountered Internal Error | The Athena connection encountered an internal error. If this error persists, please contact support. | | PRQL-GAIX | Athena Table Missing | The Athena query engine encountered an error. The table name was not recognized. | | PRQL-MZXD | Athena Table Not Found | The Athena engine could not find the table to read from or merge into. Please verify that the table exists and the connection principal has been granted sufficient permissions. | | PRQL-XYYX | Athena Type Mismatch | Athena attempted to use an operator on values of differing types. Please check the source query for type mismatches. | | PRQL-WHHO | Athena Unable To Assume Role | Athena was unable to assume the specified role. Please verify the role ARN. | | PRQL-VMEX | Athena Unsupported Timestamp Precision Error | Athena source data of type "timestamp(6)" is unsupported. Please expose the column as type "timestamp(3)". | | PRQL-QTSM | Athena Unsupported Timestamp Type Error | Athena source data of type "timestamp with time zone" is unsupported. Please expose the column as type "timestamp". | | PRQL-MDSR | Azure Authorization Expired | The Azure server returned an error indicating the access token is expired. Please check that the provided shared access key is valid and has been granted sufficient permissions. | | PRQL-BFKH | Azure Blob Snapshots Prevented Delta Table Cleanup | A file in the destination bucket could not be deleted because it has blob snapshots. Please first delete all snapshots of blobs under the table prefix, and ensure soft deletes are disabled on the storage account. Then, try again. | | PRQL-ZDJR | Azure Blob Storage Authentication Failed | Azure Blob Storage encountered an error. Please check that the provided connection principal has been granted sufficient permissions. | | PRQL-UXWZ | Azure Blob Storage Authorization Error | Azure Blob Storage returned an authorization error. Please verify your SAS token includes the required permissions on the target Container and has not expired. If network access is restricted on the storage account, ensure the service egress IP is in the allowed range. | | PRQL-GJWR | Azure Blob Storage Host Not Found | The Azure Blob Storage host could not be found. Please check that the provided host name is valid. | | PRQL-VMDR | Azure Blob Storage Invalid SAS Token | Azure Blob Storage could not authenticate the request. Please verify you have generated an SAS token (access keys are not supported), your SAS token has not expired, and was copied correctly, then try again. | | PRQL-YZQZ | Azure Databricks Storage Request Not Authorized | The Azure Databricks storage request failed authorization. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Please verify your Azure storage credentials or firewall exception settings. | | PRQL-JRKJ | BigQuery Authentication Failed | The BigQuery connection was refused. Please check that that the provided connection principal has been granted sufficient access. | | PRQL-PVER | BigQuery Authentication Failed | The BigQuery connection was refused. Please check that that the provided connection principal has been granted sufficient access. | | PRQL-SPJT | BigQuery Authentication Failed | The BigQuery server rejected the connection. Please check that the provided authentication principal and credentials are valid. | | PRQL-AVSM | BigQuery Avro File Schema Mismatch | An Avro file backing this BigQuery table is missing a column the table schema requires. Please check the pipeline writing the table's underlying files, or add a default value for the column. | | PRQL-UJSZ | BigQuery Column Already Exists | The BigQuery engine encountered an error. The target column name already exists. | | PRQL-NIKQ | BigQuery Column Does Not Exist | The BigQuery engine encountered an error. The target column name is unrecognized. | | PRQL-BJRB | BigQuery Column Rejected Null Value | A column in the destination table is marked REQUIRED but received a null value. This can occur when a column present in the destination table is missing from the write. Please contact support for more information. | | PRQL-VKGX | BigQuery Column Type Mismatch | One or more column's data type does not match the existing destination column. Please verify all destination table column types match the types defined in the corresponding model. | | PRQL-UCFB | BigQuery Connection Refused | The provided Google Cloud project has not enabled BigQuery. | | PRQL-CQEX | BigQuery Custom Query Quota Exceeded | A custom BigQuery query usage quota set by your administrator was exceeded. Please raise the quota or wait for it to reset. | | PRQL-FGWV | BigQuery Data Processing Failed | The BigQuery data transfer encountered an error during data transfer. The BigQuery target table may have duplicate data preventing successful updates. | | PRQL-JEBJ | BigQuery Dataset Not Found | The specified BigQuery dataset was not found in the configured project and region. Please verify the dataset name and region provided and try again. | | PRQL-TIRE | BigQuery Disallowed Bucket Retention Policy | Bucket and object retention policies are disallowed on BigQuery staging policies. Please remove the policy or use a bucket without a retention policy. | | PRQL-ZVKN | BigQuery GCS Authentication Error | The service account does not have sufficient access to the GCS bucket. Please review the list bucket permissions provided via IAM roles. | | PRQL-BRLC | BigQuery Invalid Bucket Region | The provided GCS bucket region is invalid. Please ensure the specified region field is exactly one of the following valid GCS bucket location names: [https://docs.cloud.google.com/storage/docs/locations](https://docs.cloud.google.com/storage/docs/locations). | | PRQL-WMRB | BigQuery Rate Limit Exceeded | One or more queries has exceeded a BigQuery rate limit. Please contact support for more information. | | PRQL-PWQD | BigQuery Runtime Encountered Error | The BigQuery connection encountered an error. | | PRQL-GKCB | BigQuery Server Encountered Error | The BigQuery API was temporarily unavailable (HTTP 503). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Cloud status page to confirm service health. | | PRQL-FGCV | BigQuery Service Account Disabled | The BigQuery connection was refused because the configured service account has been disabled. Please re-enable the service account in GCP IAM & Admin, or create a new service account and update configuration details. Then, try again. | | PRQL-AEVP | BigQuery Table Not Accessible | The user does not have permission to query the specified BigQuery Table, or it was not found in the configured region and dataset. Please review the user's permissions on the table and dataset, and the region and dataset names provided. | | PRQL-MJOZ | ClickHouse Connection Refused | The provided ClickHouse host was not found. Please check that the provided host name is valid and accessible. | | PRQL-OEVX | ClickHouse Connection Refused | The provided ClickHouse host was not found. Please check that the provided host name is valid and accessible. | | PRQL-UFTZ | ClickHouse Connection Refused | The provided ClickHouse host refused the connection. Please check that the provided host name is valid and accessible. | | PRQL-WKVZ | ClickHouse Connection Timed Out | The connection to the ClickHouse host timed out. Please check that the host and port are reachable and that the static IP address (or addresses) are allowlisted in your security group/firewall. | | PRQL-CTBL | ClickHouse Insufficient Create Table Permissions | The ClickHouse user does not have permission to create tables. Please ensure both CREATE TABLE and CREATE TEMPORARY TABLE are granted to the ClickHouse user on the target database and try again. | | PRQL-BKFC | ClickHouse Insufficient Read Permissions | The ClickHouse user does not have permission to read from the data tables and/or schema. Please ensure to grant SELECT on the database, tables for sync, and information\_schema and try again. | | PRQL-DSAJ | ClickHouse Insufficient S3 Staging Bucket List Permissions | The ClickHouse instance was denied access to the S3 staging bucket while checking whether a staged file exists before writing. Please check that you have provided the s3:ListBucket permission on the staging bucket itself (without any key prefix added), and try again. | | PRQL-WKRF | ClickHouse Insufficient S3 Write Permissions | The ClickHouse user cannot write to the S3 staging bucket. Please grant the S3 write permission to the ClickHouse user with GRANT S3 ON *.* to \@'%' (or GRANT WRITE ON S3 if your instance uses granular enable\_read\_write\_grants) and try again. | | PRQL-BNPN | ClickHouse Insufficient View Permissions | The ClickHouse user does not have permissions to create and/or drop views generated during transfers. Please grant CREATE VIEW and/or DROP VIEW to the ClickHouse user on the target database and try again. | | PRQL-SKFD | ClickHouse Invalid Connection Parameter | The provided ClickHouse credentials contain an invalid character. Please check that the username and password do not contain special characters such as @, /, or ?. | | PRQL-UQKO | ClickHouse Memory Limit Exceeded | The ClickHouse instance ran out of memory while exporting data. Please scale up the service's maximum memory limit and try again. | | PRQL-XCNS | ClickHouse Rate Limited | The Clickhouse server encountered a resource quota limit for simultaneous queries. | | PRQL-ZXEC | ClickHouse TLS Certificate Expired | The ClickHouse endpoint presented an expired TLS certificate. Please renew the certificate on the server or proxy terminating TLS and try again. | | PRQL-VCXL | Clickhouse Invalid Truncate Permissions | The Clickhouse engine encountered an error. Please check that the provided user has been granted sufficient permissions to truncate tables. | | PRQL-NJEW | Data Serialization Error | One or more rows from the source cannot be parsed. This may be caused by a row exceeding byte size limits, invalid/non-supported encoded characters, unquoted values, or column errors. Please contact support for more information. | | PRQL-BKRG | Databricks Authorization Failed | The provided service principal is not authorized to use the SQL endpoint. Please confirm that the service principal is listed as an authorized user with Can Use within the SQL Warehouse permissions, and try again. | | PRQL-AXJS | Databricks Catalog Not Found | The specified Databricks catalog was not found. Please check the catalog name in the connection configuration exists in the target workspace and try again. | | PRQL-CJFX | Databricks Catalog Permission Denied | The Databricks connection was refused due to insufficient catalog permissions. Please check that the provided connection principal has been granted USE CATALOG permission on both the target catalog and Databricks system catalog, then try again. | | PRQL-CSVN | Databricks Concurrent Write Conflict | The Databricks server encountered a concurrent update to the target table. Please ensure no other INSERT, DELETE, UPDATE, or MERGE queries are running. | | PRQL-LADF | Databricks Connection Closed Prematurely | The Databricks connection closed before the statement finished executing. Check the Databricks status page and confirm the target SQL warehouse is running and reachable. | | PRQL-MKEW | Databricks Connection Temporarily Rejected | The Databricks connection was temporarily rejected because the service is under maintenance. | | PRQL-ZQAC | Databricks Delta Table Metadata Missing From Storage | One or more Delta table transaction log files are missing from the underlying storage bucket. Please check bucket lifecycle policies to ensure no files are deleted, and contact support to learn more. | | PRQL-LRXO | Databricks Destination Table Missing Model Column | The destination table is missing a column the model expects. Please restore the column or run a full refresh to rebuild the table. | | PRQL-WBZK | Databricks Destination Table Ownership Required | The connection principal does not own the destination table. Please transfer ownership of the table to the connection principal, then retry. | | PRQL-VIDE | Databricks Duplicate Row Error | The Databricks data transfer encountered an error during data transfer. The Databricks target table or staged data may have duplicate data preventing successful updates. Please confirm there is no duplicate data in the data source, and contact support if the issue persists. | | PRQL-IPBL | Databricks IP Access List Restriction | The Databricks connection was blocked by an IP access list restriction at the Databricks account or workspace-level. Please ensure any configured IP access lists include the data sync service's static egress IP(s). | | PRQL-JMES | Databricks Identity Not Assigned To Workspace | The connection principal is not a member of the targeted Databricks workspace. Please confirm the user or service principal is assigned to that workspace and that the configured host points to the correct workspace, then try again. | | PRQL-XTZH | Databricks Insufficient Credits Available | The connection to Databricks could not be established as the Databricks account has insufficient credits. Please add a payment method or purchase additional credits. | | PRQL-OLKS | Databricks Internal Query Failure | Databricks reported an internal error while executing the query. Please check the Databricks status page and retry. | | PRQL-JZQZ | Databricks Invalid Access Token | The Databricks connection was refused due to an invalid access token. Please check that the access token (`dapi...`) is valid and the provided connection principal has been granted sufficient access. | | PRQL-OAUT | Databricks Invalid OAuth Credentials | The Databricks connection was refused because the OAuth client credentials were rejected. Please verify that the Client ID and Client Secret are correct and the credential has not expired, then please try again. | | PRQL-QOCK | Databricks Java IO Error Encountered | Databricks encountered an I/O error while executing the operation. Review the Databricks query/driver logs and the Databricks status page to identify the failure. | | PRQL-VIZF | Databricks Network Error Encountered | The connection to Databricks was dropped unexpectedly. Check the Databricks status page and confirm the target SQL warehouse is running and reachable. | | PRQL-FUOM | Databricks Operation Not Supported | The Databricks server encountered an error. Please check that the provided connection principal has been granted sufficient access. | | PRQL-QWEO | Databricks Query Timeout | The Databricks query connection timed out. If this error persists, please contact support. | | PRQL-CKSR | Databricks Rate Limit | The Databricks server encountered a rate limit. If this error persists, please contact support. | | PRQL-CYNR | Databricks Runtime Error Encountered | Databricks could not schedule the query and returned a temporary error (HTTP 503) asking to retry. This usually resolves on the next run. If the error persists, check the Databricks status page and the target SQL warehouse's health and load. | | PRQL-GMXD | Databricks Runtime Error Encountered | The Databricks server does not have Predictive Optimization enabled. | | PRQL-INXB | Databricks Serverless Warehouse Disabled | The Databricks warehouse cannot start because serverless warehouse compute is disabled in the global warehouse configuration. Please contact your Databricks administrator to enable compute for serverless warehouses, and try again. | | PRQL-YBLK | Databricks Service Unavailable | Databricks returned a 503 Service Unavailable response. This may be caused by Databricks SQL warehouse autoscaling, startup delays, warehouse load, or a transient Databricks service issue. Please check that both the target SQL warehouse and Databricks servers are healthy, and try again. | | PRQL-CMRT | Databricks Storage Authorization Failure | Databricks failed to execute one or more staging upload queries with PUT. If using Unity Catalog, please ensure the Write Volume permission has been provided to the Service Principal or user. If using Hive Metastore, please ensure write and delete permissions have been provided on the object storage location. | | PRQL-BUMS | Databricks Table Or View Not Found | A referenced table or view was not found. Please check that the target tables were not dropped or renamed. | | PRQL-PSZD | Databricks Warehouse Stopped | The Databricks transfer failed because the target warehouse was stopped. | | PRQL-YWEX | Databricks Workspace Rate Limit Exceeded | The Databricks workspace exceeded its API rate limit. Please retry later, reduce concurrent requests, and/or contact Databricks to increase request limits in the workspace. | | PRQL-DBSH | Databricks no such host | Connection to Databricks could not be established on failure to reach host. Please ensure that the provided hostname is accurate and the Databricks cluster is publicly accessible. | | PRQL-DLDQ | Delta Lake Duplicate Row Error | One or more duplicate rows have been detected in the source data. Please ensure that each row's primary key is unique/de-duplicated and retry. | | PRQL-WKXN | Delta Lake Engine Stopped Unexpectedly | The Delta Lake engine stopped unexpectedly during the transfer. Please retry the transfer. If this error persists, please contact support. | | PRQL-UFQE | Delta Lake Object Not Found | The Delta Lake engine could not find an object. Make sure the object exists and the writer has read access to the bucket. | | PRQL-GFIS | Delta Lake Runtime Error | The Delta Lake engine encountered an error. | | PRQL-CISE | Delta Lake Table Creation Failed | The delta lake table could not be created due to existing conflicting metadata in the bucket prefix. | | PRQL-GTRI | Delta Lake Transaction Log Commit Failed | The transaction log could not be committed to the destination bucket. This is most commonly caused by failing to disable soft delete defaults on the storage account. Please first check that both blob and container soft deletes are disabled on the storage account, then ensure no other engines write to the table concurrently before retrying. | | PRQL-BERH | Destination Table Not Found | The destination table was not found in the target schema. Please ensure the destination table exists at the configured schema and with the target table name. Contact support for more information. | | PRQL-DOOM | DuckDB Out of Memory | The transfer worker ran out of memory. If this error persists, please contact support. | | PRQL-PZCR | Federated Token Exchange Rejected | Google rejected the identity provider's token during the workload identity federation exchange. Verify the workload identity pool and provider settings, including the allowed audience, and that the identity provider issues JWT access tokens. | | PRQL-QGBC | GCS Bucket Not Found | A GCS bucket under the configured bucket name and region could not be found in the target GCP project. Please check that the name and region are set correctly, create the bucket if it does not already exist, then try again. | | PRQL-SLVK | GCS Missing Bucket Storage GET Permission | The GCS server encountered an error due to insufficient permissions. Please check that the provided connection principal has been granted 'storage.buckets.get'. | | PRQL-ZQZQ | GCS Missing Create Permission | The GCS server encountered an error due to insufficient permissions. Please check that the provided connection principal has been granted 'storage.objects.create'. | | PRQL-AYSD | GCS Missing Delete Permission | The GCS server encountered an error due to insufficient permissions. Please check that the provided connection principal has been granted 'storage.objects.delete'. | | PRQL-KJCH | GCS Server Encountered Error | Google Cloud Storage returned an internal server error (HTTP 500). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Cloud status page to confirm service health. | | PRQL-NXPY | GCS Server Encountered Error | Google Cloud Storage was temporarily unavailable (HTTP 503). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Cloud status page to confirm service health. | | PRQL-IXOL | Google Sheets Authentication Failed | The Google Sheets server refused the connection. | | PRQL-NEFP | Google Sheets Authentication Failed | The Google Sheets server refused the connection. | | PRQL-VDJX | Google Sheets Data Processing Failed | The Google Sheets server encountered a temporary error and the connection was dropped. If this error persists, please contact support. | | PRQL-LNWT | Google Sheets Host Not Found | The Google Sheets host could not be found. Please check that the provided host name is valid. | | PRQL-CELL | Google Sheets Overloading Error | A Google Sheet may not have more than 10,000,000 cells. If the error persists, please choose another destination type. | | PRQL-BOSR | Google Sheets Server Encountered Error | Google Sheets returned a bad gateway error (HTTP 502). This is typically a temporary issue and resolves automatically. | | PRQL-CMET | Google Sheets Server Encountered Error | Google Sheets was temporarily unavailable (HTTP 503). This is typically a temporary issue and resolves automatically. If the error persists, check the Google Workspace status page to confirm service health. | | PRQL-GPDB | Google Sheets Server Encountered Error | Google Sheets returned an internal server error (HTTP 500). This is typically a temporary issue and resolves automatically. | | PRQL-YICW | Google Sheets Server Encountered Error | The Google Sheets server encountered a resource quota limit. | | PRQL-ICED | Iceberg Authorization Failed | A query failed due to insufficient database permissions. Please check that the provided connection details are correct and the principal has been granted sufficient permissions to the target catalog and object store. | | PRQL-FJVY | Identity Provider Rejected Client Credentials | The identity provider rejected the configured OAuth client credentials (invalid\_client). Verify the client ID and client secret and try again. | | PRQL-BVHD | Identity Provider Token Fetch Failed | Prequel could not fetch an OAuth token from the configured identity provider token endpoint. Verify the token endpoint URL is publicly reachable over https and that it accepts the configured client credentials. | | PRQL-BBGU | Import Artifact Staging Error | Prequel could not stage the batch file for delivery. The records were not sent and will be retried on the next load. | | PRQL-NSCH | Import Dataset Schema Unavailable | The dataset schema is unavailable until an extract has successfully extracted rows for this dataset. | | PRQL-MLAI | Import Delivery Outcome Integrity Error | Prequel detected an internal inconsistency while recording delivery results. The affected records were not marked delivered and will be retried on the next load. Please contact support if this persists. | | PRQL-QPMF | Import Destination Bad Gateway | The destination endpoint returned HTTP 502. Undelivered records are retried on the next scheduled load. | | PRQL-JPDQ | Import Destination Bad Request | The destination endpoint rejected the record with HTTP 400. Check the record data and destination configuration. | | PRQL-IMEK | Import Destination Client Error | The destination endpoint rejected the record with a client error (4xx). Check the destination configuration and record data. | | PRQL-KTQN | Import Destination Connection Refused | The destination endpoint refused the connection. Verify the URI and that the endpoint is accepting connections. | | PRQL-YCHU | Import Destination Context Cancelled | The delivery operation was cancelled before all records could be delivered. Undelivered records will be retried on the next load cycle. | | PRQL-FVRM | Import Destination DNS Error | DNS resolution failed for the destination endpoint. Verify the hostname in the destination URI. | | PRQL-CNJK | Import Destination Forbidden | The destination endpoint returned HTTP 403. Verify webhook signature configuration and credentials. | | PRQL-PRHQ | Import Destination Gateway Timeout | The destination endpoint returned HTTP 504. If the response carried a Retry-After header the request was retried immediately, otherwise the record will be retried on the next scheduled load. | | PRQL-QFJW | Import Destination Headers Too Large | The destination endpoint returned HTTP 431. Reduce the number or size of custom headers in the request template. | | PRQL-PHOK | Import Destination Internal Server Error | The destination endpoint returned HTTP 500. Undelivered records are retried on the next scheduled load. If the error persists, check the destination endpoint health. | | PRQL-TWER | Import Destination Invalid Acknowledgement | The destination acknowledgement payload was malformed or failed schema validation. | | PRQL-HWLT | Import Destination Network Error | A network error occurred while delivering a record to the destination endpoint. Check the destination URI and network connectivity. | | PRQL-VFTB | Import Destination Not Found | The destination endpoint returned HTTP 404. Verify the URI in the request template. | | PRQL-QAHC | Import Destination Payload Too Large | The destination endpoint returned HTTP 413. Reduce the record size or adjust the destination's payload limits. | | PRQL-WQKF | Import Destination Rate Limited | The destination endpoint rate-limited the request (HTTP 429). If the response carried a Retry-After header the request was retried immediately, otherwise the record will be retried on the next scheduled load. | | PRQL-EZDQ | Import Destination Request Error | Failed to construct the HTTP request for the destination endpoint. Check the destination configuration. | | PRQL-TNXG | Import Destination Server Error | The destination endpoint returned a server error (5xx). Undelivered records are retried on the next scheduled load. If the error persists, check the destination endpoint health. | | PRQL-HNNX | Import Destination Service Unavailable | The destination endpoint returned HTTP 503. If the response carried a Retry-After header the request was retried immediately, otherwise the record will be retried on the next scheduled load. | | PRQL-XNWB | Import Destination TLS Error | TLS handshake failed with the destination endpoint. Verify the endpoint supports HTTPS and has a valid certificate. | | PRQL-BXPM | Import Destination Template Error | The destination request template is misconfigured. Verify that the body template syntax is valid and all referenced fields exist in the destination's record schema. | | PRQL-GMVJ | Import Destination Timeout | The request to the destination endpoint timed out. Undelivered records are retried on the next scheduled load. If the error persists, check the destination endpoint latency. | | PRQL-NROT | Import Destination Unprocessable Entity | The destination endpoint returned HTTP 422. The record data is syntactically valid but semantically incorrect. | | PRQL-NBSI | Import Job Cancelled | The import job was cancelled. Please retry — or contact support if you did not initiate this and the issue persists. | | PRQL-VXPK | Import Primary Key Not String | The primary key column must be of type string (text/varchar). Please ensure the source table primary key column is string-typed. | | PRQL-KVZK | Import Provider Not Configured | No enabled import provider is available for this request. Create one with `POST /import/providers` or enable an existing provider before creating datasets, extracts, or loads. | | PRQL-WCKH | Import Transform Invalid Configuration | A stream mapping configuration was invalid or missing required fields like 'name' or 'target\_field'. | | PRQL-BKDX | Import Transform Invalid Payload | The transformation pipeline encountered an invalid JSON payload. Please ensure that stream records are valid JSON. | | PRQL-TRUN | Import Transform Runtime Error | The transformation pipeline encountered one or more runtime errors. Please review the mapping expressions and input data. | | PRQL-NOTD | Import Transform Unsupported Mapping | An unsupported mapping type was used in stream configuration. Supported types are 'field' and 'expr'. | | PRQL-VKHA | Import Transform Validation Failed | A stream record failed validation against the destination record schema after transformations were applied. | | PRQL-OHHG | Import Worker Unavailable | No import worker is currently available to process this request. The system will auto-recover shortly. Please retry. | | PRQL-INUM | Invalid Input: Numerical Type | One or more values could not be cast to a numerical type. Please ensure all source data columns of numerical types contain valid numerical values. Contact support if this error persists. | | PRQL-FGAD | Invalid Timestamp Value in Source Column | An invalid timestamp value was detected in a source column. Please review your timestamp columns for invalid values. | | PRQL-XTQZ | Missing Datalake Cache Resource | The dataset's cached data was not found in its datalake. Ensure the dataset belongs to a provider whose datalake bucket remains active and intact, and try again. | | PRQL-CLNU | MongoDB Authentication Failed | The MongoDB connection was refused due to insufficient permissions. Please check that the provided connection principal has been granted sufficient access. | | PRQL-KWZN | MongoDB Authentication Failed | The MongoDB connection was successful, but the user is not authorized. Please check that the provided user has access. | | PRQL-MJUM | MongoDB Authentication Failed | The MongoDB connection was refused. Please check that the provided MongoDB access details are correct. | | PRQL-FBNN | MongoDB Collection Not Found | The provided MongoDB collection was not found. | | PRQL-BGZS | MongoDB Connection Failed | The MongoDB connection was unsuccessful due to an unexpected configuration error. If the error persists, please contact support. | | PRQL-KDCW | MongoDB Connection Refused | The MongoDB connection was refused. Please check that the provided MongoDB server is valid and accessible. | | PRQL-MDXE | MongoDB Runtime Encountered Error | The MongoDB connection exceeded a space quota. | | PRQL-VIPD | MongoDB Runtime Encountered Error | The MongoDB connection encountered a memory limit. | | PRQL-OPXQ | MotherDuck Duplicate Primary Key Error | The transfer could not be completed because the source data contains more than one row with the same primary key value for this model. Please check the source table duplicate primary key values, deduplicate if needed, and try again. | | PRQL-QYYA | MySQL Authentication Failed | The MySQL connection was refused due to insufficient permissions. Please check that the provided connection details have been granted sufficient access. | | PRQL-AZBE | MySQL Connection Failed | The provided MySQL host was not found. Please check the the provided host name is valid and accessible. | | PRQL-DQTJ | MySQL Connection Failed | The provided MySQL port is invalid. | | PRQL-EWAS | MySQL Insufficient Create Table Permissions | The writer user does not have sufficient permissions to create tables in the destination schema. Please grant CREATE on the destination schema to the configured user and try again. | | PRQL-VMTE | MySQL Insufficient User Permissions | One or more permissions on the database are missing from the writer user. Please ensure all required permissions are provided and retry. | | PRQL-WCLX | MySQL Invalid Syntax Error | A SQL syntax error was encountered. Please contact support for more information. | | PRQL-AREW | Object Storage Connection Failed | The specified bucket name contains invalid characters. | | PRQL-EGJN | Object Storage Connection Failed | The specified bucket does not exist. | | PRQL-SAGK | Object Storage List IO Timeout | Object Storage client encountered IO timeout listing bucket objects. If this error persists, please contact support. | | PRQL-NQFC | Oracle Authentication Failed | The provided Oracle credentials are invalid. Please check that the correct username and password have been provided and try again. | | PRQL-WPMU | Oracle Connection Failed | The provided Oracle port is invalid. Please check that the provided port is a valid TCP port (typically 1521). | | PRQL-VRDH | Oracle Database Not Found | The Oracle listener at the provided host does not recognize the configured database. Please check that the provided database name is correct and try again. | | PRQL-OXKB | Oracle Host Not Found | The provided Oracle host was not found. Please check that the provided host name is valid and accessible. | | PRQL-CIUS | Oversized Decimal Value | Encountered a value of type DECIMAL that is too large to be converted to DECIMAL(38,19). | | PRQL-MCOQ | Postgres Authentication Failed | The Postgres connection was refused. Please check that the provided connection details are correct. | | PRQL-JXOS | Postgres Conflict With Recovery | The Postgres connection was dropped by the server. This is usually due to a Hot Standby related query timeout. If this error persists, please contact support. | | PRQL-RXOI | Postgres Connection Failed | The Postgres connection was refused. Please check that the provided host is valid and accessible. | | PRQL-TPOQ | Postgres Connection Failed | The Postgres connection was refused. Please check that the provided host is valid and accessible. | | PRQL-VLMV | Postgres Connection Failed | The Postgres connection was refused. Please check that the provided host is valid and accessible. | | PRQL-QJXV | Postgres Data Processing Failed | The Postgres connection encountered an error. At least one expected column was not found. | | PRQL-SMHC | Postgres Data Processing Failed | The Postgres connection encountered an error. At least one expected relation was not found. | | PRQL-XLSX | Postgres Data Processing Failed | The Postgres data transfer encountered duplicate primary key values. Please check that the primary key column values are unique for this recipient. | | PRQL-FLQR | Postgres Missing PostGIS Extension | The PostGIS extenstion is required for the Postgres to recognize the type `geometry`. Please make sure PostGIS is installed. | | PRQL-KKIM | Postgres Runtime Encountered Error | The Postgres connection encountered an unexpected error and was dropped. If this error persists, please contact support. | | PRQL-KPFZ | Postgres Runtime Encountered Error | The Postgres connection was dropped by the server. If this error persists, please contact support. | | PRQL-OKZP | Postgres Runtime Encountered Error | The Postgres connection was refused on the schema. Please check that the provided connection principal has been granted sufficient access. | | PRQL-UKMC | Postgres Runtime Encountered Error | The Postgres connection was dropped due to a conflict with the recovery instance. This is usually due to a Hot Standby related query timeout. If this error persists, please contact support. | | PRQL-ZSRS | Postgres Runtime Encountered Error | The Postgres connection was refused on the view. Please check that the provided connection principal has been granted sufficient access. | | PRQL-PGST | Postgres Statement Timeout Exceeded | The query exceeded the statement\_timeout configured on the Postgres database. Please raise statement\_timeout for the connecting role/user: [https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STATEMENT-TIMEOUT](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-STATEMENT-TIMEOUT). | | PRQL-ROTX | Postgres cannot execute write in read-only transaction | Postgres is unable to execute the write query. Please ensure that Postgres is configured upon a writer endpoint and with write permissions. | | PRQL-QDFS | Postgres table missing replica identity | Postgres is unable to update a destination table because it is part of a logical replication publication but has no replica identity configured. Please ensure that any published tables have a primary key, or are set to REPLICA IDENTITY FULL, or exclude them from the publication. | | PRQL-RSLM | Record Size Limit Exceeded | One or more records in the source data exceed the maximum allowed size for import. Please reduce the size of these records or contact support for assistance. | | PRQL-DXHS | Redshift Authentication Failed | The provided Redshift connection was refused due to insufficient permissions. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-RSFA | Redshift Authentication Failed | The Redshift connection was refused. Please check that the provided connection principal has been granted sufficient access. | | PRQL-BOWD | Redshift Authorization Failed | The provided Redshift query failed due to insufficient database permissions. Please check that the provided connection details are correct and the principal has been granted sufficient permissions to the target database. | | PRQL-XKWZ | Redshift Cannot Drop Table with Dependencies | A DROP table operation failed due to one or more downstream consumers dependent on it. Please temporarily revoke access from downstream consumers to the destination table for the full refresh transfer to succeed. | | PRQL-NFCL | Redshift Cluster Not Found | The specified Redshift cluster could not be found in the target region. Please check that the cluster identifier is correct and the staging bucket is located in the same AWS region as the Redshift cluster. | | PRQL-WKQZ | Redshift Concurrent Write Conflict | A Redshift write aborted due to a conflict with a concurrent transaction on the same table. Please ensure only a single data export writer targets this schema and no other workloads are attempting to update the table. | | PRQL-BLHI | Redshift Connection Failed | The Redshift connection was refused. Please check that the provided host is valid and accessible. | | PRQL-FTNX | Redshift Connection Failed | The Redshift connection was refused. Please check that the provided host is valid and accessible. | | PRQL-ZJSM | Redshift Connection Reset By Peer | The Redshift connection was reset. This is likely due to a network issue. Please check your SSH, firewall, and network settings. If the issue persists, please contact support. | | PRQL-CSNQ | Redshift Data Processing Failed | The Redshift connection encountered an unknown error, contact support. | | PRQL-JCVI | Redshift Data Processing Failed | The Redshift connection encountered an error. At least one expected column was not found. | | PRQL-KMNN | Redshift Data Processing Failed | The Redshift connection encountered an error. At least one expected relation was not found. | | PRQL-XVKM | Redshift IAM Role Token Audience Mismatch | AWS rejected the federated identity token because its audience did not match what the IAM role expects. Please verify the oaud condition in the role's trust policy matches the provided value. | | PRQL-WUZB | Redshift Runtime Encountered Error | The Redshift connection encountered an unexpected error and was dropped. If this error persists, please contact support. | | PRQL-RSHA | Redshift Unable to Assume IAM Role | The Redshift connection was denied permission to assume the configured IAM role. Please verify the role's trust policy allows sts:AssumeRoleWithWebIdentity from the service identity and try again. | | PRQL-MWKP | S3 IAM Role Token Audience Mismatch | AWS rejected the federated identity token because its audience did not match what the IAM role expects. Please verify the oaud condition in the role's trust policy matches the provided value. | | PRQL-HMCK | S3 Invalid AWS Role ARN | The provided AWS IAM Role ARN is invalid. Please verify the ARN format and value is correct and try again. | | PRQL-BKDC | S3 Invalid Hostname | The connection contains an invalid hostname. Please verify your bucket region is a valid AWS region code (e.g. eu-west-1). | | PRQL-JSZM | S3 Missing Delete Permission | Unable to upload file to S3. Please check that the provided role has been granted s3:DeleteObject. | | PRQL-GWUV | S3 Missing List Bucket Permission | Unable to list objects in the S3 destination bucket. Please grant the s3:ListBucket permission to the configured role on the bucket. | | PRQL-RTTA | S3 Missing Upload Permission | Unable to upload file to S3. Please check that the provided role has been granted s3:PutObject. | | PRQL-CIAC | S3 Tables Missing Get Table Metadata Location Permission | A query failed due to a missing s3tables:GetTableMetadataLocation permission in the user/role policy. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-QLDD | S3 Tables Missing Get Table Permission | A query failed due to a missing s3tables:GetTableBucket permission in the user/role policy. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-GXDD | S3 Tables Missing Update Table Metadata Location Permission | A query failed due to a missing s3tables:UpdateTableMetadataLocation permission in the user/role policy. Please check that the provided connection details are correct and the principal has been granted sufficient permissions. | | PRQL-QGQQ | S3 Unable to Assume IAM Role | The S3 connection was denied permission to assume the configured IAM role. Please verify the custom trust policy on the role allows sts:AssumeRoleWithWebIdentity from the service identity, and that the required oaud/sub conditions are set correctly, and try again. | | PRQL-AWJZ | SFTP Connection Lost | SFTP client experienced a connection loss. Please review any idle timeout or session limits configured on the SFTP server and retry. | | PRQL-ANLW | SQL Server Connection Could Not Be Established | The SQL Server connection timed out. Please verify the host is reachable, firewall rules and/or security group allows access by whitelisting the static egress IP, and the port is reachable. | | PRQL-GZDZ | SQL Server Connection Failed | The provided SQL Server host was not found. Please check that the provided host name is valid and accessible. | | PRQL-VKDT | SQL Server Data Processing Failed | The SQL Server data transfer encountered duplicate primary key values. Please check that the primary key column values are unique for this recipient. | | PRQL-UXPW | SQL Server Invalid Username/Password Characters | The transfer user's username or password contains special characters that cannot be parsed during connection. Please update the user's credentials and try again. | | PRQL-LCNW | SQL Server Size Quota Exceeded | The database has reached its size quota. Please check the database service tier storage limit and any maximum dataset size configured, and increase allocations as needed. | | PRQL-MURC | SQL Server unable to resolve collation conflict | A collation conflict cannot be resolved on the SQL Server table. Please contact support for more information. | | PRQL-CLDD | Schema Not Found Error | A query failed due to a missing schema. Please check that the specified schema exists in the catalog or metastore. | | PRQL-ZPLL | Service Temporarily Unavailable | Prequel's import service is temporarily unavailable. Please retry shortly. If the issue persists, contact support. | | PRQL-FKDS | Snowflake Authentication Error | Snowflake user access is disabled. Please contact the Snowflake admin to get more information. | | PRQL-VKCP | Snowflake Authentication Failed | The Snowflake engine refused the operation due to insufficient permissions. Please check that the provided principal has been granted the required permissions. | | PRQL-ZBSP | Snowflake Authentication Failed | The provided Snowflake credentials are invalid. Please check that the correct credentials have been provided. | | PRQL-BJKE | Snowflake Authentication Policy Rejected | Snowflake rejected the login because the current authentication policy does not allow the attempted authentication method. Ensure the user/account policy allows DRIVERS and the chosen authentication method (PASSWORD or KEYPAIR). | | PRQL-IGRW | Snowflake Authentication Rate Limited | The Snowflake connection was rejected. This is likely due to too many failed connection attempts. Please check that the correct credentials have been provided. | | PRQL-EXLI | Snowflake Connection Failed | IP whitelisting has not been properly configured in the provided Snowflake account. Please contact support to confirm the correct IP address to whitelist within Snowflake. | | PRQL-HFCO | Snowflake Connection Failed | The provided Snowflake Account Host is not valid. Please check that your provided host name is a valid Snowflake account URL (ending in 'snowflakecomputing.com'). | | PRQL-PKAN | Snowflake Data Processing Failed | The Snowflake connection has no active warehouse. Please ensure the provided user has a default warehouse configured. | | PRQL-AFWL | Snowflake Database Does Not Exist or Not Authorized | The connection attempt failed due to an issue with permissions. Please confirm that the provided Snowflake database exists and the provided user has been authorized. If the error persists, please confirm that the provided user has been successfully granted access to the designated role, and that the default\_role has been successfully configured on the user. | | PRQL-NSOE | Snowflake Duplicate Row Error | The Snowflake data transfer encountered an error during data transfer. The Snowflake target table or staged data may have duplicate data preventing successful updates. Please confirm there is no duplicate data in the data source, and contact support if the issue persists. | | PRQL-BIEM | Snowflake Host Does Not Exist | The Snowflake host was unreachable. Please ensure the host was properly entered. | | PRQL-WLAM | Snowflake Incorrect Username or Password | The provided Snowflake username or passsword is incorrect. Please check that the correct username and password have been provided. | | PRQL-MVKR | Snowflake Integer Overflow | A value in a Snowflake integer column exceeds the maximum supported 64-bit integer size. | | PRQL-JWCP | Snowflake Invalid Account Setting | Failed to read the information schema due to casing. If the Snowflake account has the setting QUOTED\_IDENTIFIERS\_IGNORE\_CASE enabled, make sure it is set to FALSE on the data transfer user. | | PRQL-COSD | Snowflake Invalid Identifier | A Snowflake query tried to reference a table or column that does not exist. If the table or column has been deleted, it may need to be recreated manually or via full refresh. Please contact support for more information. | | PRQL-OFWE | Snowflake Key Auth Error | The Snowflake connection was refused. Please confirm that the provided Snowflake public key has been successfully assigned to the Snowflake user. | | PRQL-AJHJ | Snowflake Multi-factor Authentication is Required | Multi-factor authentication is currently required for the authorized user and MFA is not supported with data syncing. Please ensure your user is of type SERVICE instead of HUMAN and try again. | | PRQL-VIED | Snowflake No Such Host Error | The Snowflake connection failed to initialize, the host provided does not exist. | | PRQL-YWUX | Snowflake Object Verification Error | The Snowflake server was unable to verify an object. Please check that the Snowflake principal has access to the required objects. | | PRQL-UPSD | Snowflake Password Expired | The given Snowflake password has expired. Snowflake suggests updating the password via the Snowflake web console. | | PRQL-QZQZ | Snowflake Quota Exceeded | The designated Snowflake warehouse cannot be resumed because the warehouse's resource monitor has exceeded its credit quota. | | PRQL-IRPW | Snowflake Runtime Encountered Error | The Snowflake account has run out of virtual warehouse trial credit. | | PRQL-JWOD | Snowflake Runtime Encountered Error | Snowflake reported that the statement was no longer executing (error 000605). This usually means the query was canceled before it could run, which is typically transient and resolves automatically. If the error persists, check whether the target warehouse is overloaded and scale it up if queries are queuing. | | PRQL-WTKQ | Snowflake Runtime Encountered Error | The Snowflake account has been suspended due to the lack of a payment method. | | PRQL-SSTE | Snowflake Session Token Expired | The Snowflake session token has expired. Please ensure the configured warehouse is active and not overloaded, and retry the transfer. | | PRQL-SNOA | Snowflake Stage is Not Authorized | The connection encountered an issue accessing a Snowflake stage. If you have existing FUTURE STAGE grants at the database or schema level, please ensure that the transfer user's default role is provided GRANT READ and GRANT WRITE ON FUTURE STAGES in the target schema. | | PRQL-DSIO | Snowflake Timeout | Snowflake client encountered statement or warehouse timeout and canceled query. | | PRQL-BKEK | Snowflake Unknown Internal Error | Snowflake returned an internal error. Review the failed query in Snowflake's query history and check the Snowflake status page to determine whether the issue is ongoing. | | PRQL-DICX | Snowflake Warehouse Encountered Permissions Error | The Snowflake warehouse encountered an error. Please check that the provided service account has been granted sufficient permissions. Specifically, please check for permissions on existing tables and schemas. | | PRQL-KFSS | Source Circuit Breaker Tripped | The job encountered too many source mapping, transform, or validation errors and triggered the circuit breaker. Execution was halted early to protect system resources. Please review the failed records to determine the root cause. | | PRQL-DCBT | Stream Circuit Breaker Tripped | Deliveries for this stream failed persistently and its circuit breaker was tripped. Loads for this stream will be skipped until the circuit breaker is reset, which you can do by updating the stream once the underlying failure is addressed. | | PRQL-ANCT | Transfer Dispatch Error | The transfer failed to be dispatched to a worker. | | PRQL-IDWD | Trino Exceeded Partitions Limit | The Trino engine encountered an error because too many partitions were found. | | PRQL-ERMS | Unknown Error | An unknown error occurred. | | PRQL-XFVC | Unsupported Table Type | The specified table type is not supported for this datalake operation. | # Reporting Source: https://docs.prequel.co/import/logging/reporting Generate and view reports for your Prequel Import data pipelines The Prequel Import Admin UI is designed to support most reporting and observability workflows. For advanced users, this data is also available via API or as a live export to your data platform of choice. ### Reporting APIs Content coming soon. ### Reporting Export To setup Reporting Export for your account, please contact your Prequel Representative. # Webhooks and monitoring Source: https://docs.prequel.co/import/logging/webhooks-and-monitoring Set up webhooks and monitoring for your Prequel Import data pipelines Prequel Import emits webhooks for a number of event types. You can subscribe to specific event types and configure delivery through HTTPS, Slack, PagerDuty, or Datadog using the `/webhooks` endpoints. ## Managing webhook subscriptions The webhook subscription API supports full CRUD: | Method | Path | Purpose | | -------- | ------------------------ | --------------------------------------- | | `GET` | `/webhooks` | List webhook subscriptions. | | `POST` | `/webhooks` | Create a webhook subscription. | | `GET` | `/webhooks/{webhook_id}` | Retrieve a single webhook subscription. | | `PATCH` | `/webhooks/{webhook_id}` | Update a webhook subscription. | | `DELETE` | `/webhooks/{webhook_id}` | Delete a webhook subscription. | Each webhook subscription targets exactly one delivery vendor. The request body uses a `webhook` wrapper containing vendor-specific options. ## Webhook event types Prequel Import webhook event types are in flux and are expected to change. They will be documented here soon. ## Delivery methods ### HTTP POST and GET Prequel Import supports HTTPS callbacks to your webhook receiver. When creating a webhook with `vendor: generic_post`, payloads arrive as JSON. The `vendor: generic_get` type delivers payloads as URL parameters. Both require a `url` and accept an optional `api_key` for authenticated receivers. ### Third-party integrations You can route events to PagerDuty (`vendor: pagerduty`), Slack (`vendor: slack`), and Datadog (`vendor: datadog`) with vendor-specific payload formatting. Slack expects a `https://hooks.slack.com/services/...` URL. PagerDuty and Datadog authenticate with their respective integration keys passed as `api_key`. ## Authentication Webhooks can include an API key for destinations requiring authentication. Prequel signs every payload and includes the signature in the `X-Prequel-Webhook-Signature` header for verification purposes. ## Versioning At this time, webhook versioning is in flux and is expected to change. ## Payload structure ### Headers | Header | Description | | ----------------------------- | -------------------------------------- | | `Content-Type` | Always `application/json` | | `X-Prequel-Webhook-Timestamp` | Event send timestamp | | `X-Prequel-Webhook-Signature` | SHA-256 RSA PKCS1 v1.5 signature | | `X-Prequel-Webhook-Digest` | Optional SHA-256 hash for verification | ### Body format All events follow this structure: ```json title="Webhook payload" icon="brackets-curly" expandable theme={null} { "type": "resource_type.event_type", "version": "XXXX-XX-XX", "created_at": "...", "data": { // event-specific content } } ``` ## Signature verification Prequel uses asymmetric cryptography with RSA key pairs. The private key signs payloads; your account's public key verifies authenticity. ### Verification steps **1. Retrieve Public Key** Access your webhook public key via the `/public/signatures/webhook-public-key` API endpoint. Fetch this value rather than hardcoding it, since the key may be rotated. **2. Reconstruct Signing Data** Extract the timestamp from the `X-Prequel-Webhook-Timestamp` header (RFC 3339 format). Combine timestamp, a period (`.`), and the raw JSON body. Hash this concatenated string using SHA-256. Use the raw request body before JSON deserialization, as parsing may introduce subtle changes. **Validate Body Hash** The `X-Prequel-Webhook-Digest` header contains the SHA-256 hash of the raw body only. Compare this against your computed hash to verify correct body handling. Do not use this for signature confirmation. **3. Confirm Signature** Verify the signing data hash against the signature in `X-Prequel-Webhook-Signature` using your public key with PKCS1 v1.5 scheme. Hex-decode Prequel's signature before comparison. **4. Check Timestamp** Implement a time window (e.g., 5 minutes) to reject outdated events and prevent replay attacks. ## Object storage manifest signatures When data is delivered to an object storage datalake (S3, GCS, ABS, S3-compatible), Prequel writes a signed manifest alongside the data so consumers can verify the manifest came from Prequel. Fetch the verification key with `GET /public/signatures/object-storage-manifest-public-key` and treat it the same way as the webhook public key. Fetch the value dynamically rather than hardcoding it, since the key may rotate. # Data storage Source: https://docs.prequel.co/import/security/data-storage Understand how Prequel Import handles and stores your data ## Data in transit Data in flight is always encrypted. Pipeline data is processed on Prequel servers for transformation and mapping, but this data is never persisted on Prequel servers longer than the lifecycle of the transfer batch. ## Data at rest As a design principal, Prequel Import never stores Provider data on Prequel servers (in any deployment model). Instead, to support efficient replication and change detection, a secure cache is maintained in your cloud. For more details, refer to the [Datalake](/import/core-concepts/datalake) documentation. # Deployment Source: https://docs.prequel.co/import/security/deployment Explore deployment options for Prequel Import including cloud and on-premise ## Deployment options Prequel Import supports multiple deployment options to meet your security and compliance requirements: * **Cloud Hosted (Managed)** - Prequel manages the infrastructure in our cloud environment * **Private Cloud Hosted (Managed)** - Prequel manages the infrastructure in an isolated cloud environment (AWS or GCP) * **Self Hosted** - You manage the infrastructure in your own environment (AWS, GCP, Azure, or metal), including across multiple parallel deployments (e.g., to satisfy data residency requirements). ## Cloud hosted ### Static IP addresses If you need to allowlist Prequel Import's IP addresses, use the following based on your region: #### US static IP Addresses ``` 35.192.85.117 ``` #### EU static IP Addresses ``` 104.199.49.149 ``` # Overview Source: https://docs.prequel.co/import/security/overview Learn about Prequel Import's security practices and compliance certifications # Security and Compliance Overview Security is built into our platform at every step and tested rigorously. Choose the [deployment model](/import/security/deployment) and access controls that fit your security requirements. For the most detailed and up-to-date information on security at Prequel, visit our [security page](https://prequel.co/security). To request a copy of our SOC 2 report, please contact your Prequel representative. # Azure blob storage Source: https://docs.prequel.co/import/sources/azure-blob-storage Connect your Azure Blob Storage container as a source This connection authenticates with a shared access signature (SAS) token scoped to the container you want to read from. The token grants read and list access only, and you can set an expiry date and revoke it at any time from your storage account settings. ## Step 1: create an access token In the Azure portal, navigate to the **Storage accounts** service and click the account that contains the container you want to read from. Make a note of the **storage account name** and the **container name**. In the navigation pane, under "Security + networking", click **Shared access signature**. Update the required accessible services and permissions: 1. Under "Allowed services", select **Blob**. 2. Under "Allowed resource types", select **Container** and **Object**. 3. Under "Allowed permissions", select **Read** and **List**. Select a "Start and expiry date/time" based on your security posture (for example, set the expiration date 6 months into the future), and click **Generate SAS and connection string**. Make a note of the **SAS token** that is generated. ![Generate SAS token](https://storage.googleapis.com/prequel_docs/images/azure-setup-details.png) If you limit network access on the storage account, add the static IP to the allowed address range so we can reach the container. Reach out to your account representative for the static IP address to use. ## Step 2: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **storage account name** from Step 1. 3. The **container name** from Step 1. 4. The **Storage account SAS token** from Step 1. # BigQuery Source: https://docs.prequel.co/import/sources/bigquery Connect your BigQuery data warehouse as a source This connection relies on two service accounts. You create a service account in your own GCP project that holds the BigQuery read permissions described below. Our service account is then granted permission to impersonate yours using short-lived tokens. As a result, no private key is ever shared, every action appears in your own audit logs, and you can revoke access at any time through your IAM settings. The recommended authentication method, service account impersonation, is covered in Step 3. If your policy requires a different approach, that step also describes how to use a service account key. ## Step 1: create a service account In the GCP console, in the same project as your BigQuery data, navigate to the **IAM & Admin** menu, open the **Service Accounts** tab, and click **Create service account**. ![Create service account menu](https://storage.googleapis.com/prequel_docs/images/gcp-create-service-account-menu.png) Give the service account a descriptive name and click **Create and Continue**. ![Service account name options](https://storage.googleapis.com/prequel_docs/images/gcp-service-account-name-options.png) Grant the new service account the **BigQuery User** role. This allows it to run the read queries that extract your data. ![Grant BigQuery User role](https://storage.googleapis.com/prequel_docs/images/gcp-bigquery-user.png) Click **Done** to finish creating the account. Open the service account in the list and make a note of its **email**, which looks like `service-account-name@project-id.iam.gserviceaccount.com`. You will provide this email in Step 5. ## Step 2: grant read access to the source dataset In **BigQuery**, click on the dataset you want to read from. In the menu on the right, click **Sharing** and then click **Permissions**. ![BigQuery dataset sharing permissions](https://storage.googleapis.com/prequel_docs/images/bigquery-sharing-permissions.png) Click **Add Principal**. ![BigQuery add principal](https://storage.googleapis.com/prequel_docs/images/bigquery-add-principal.png) Add the service account you created in Step 1, grant it the **BigQuery Data Viewer** role, and click **Save**. ![BigQuery Data Viewer role](https://storage.googleapis.com/prequel_docs/images/bigquery-user-data-viewer.png) ## Step 3: authorize authentication Choose the authentication method that fits your security policy. Service account impersonation is recommended because it avoids sharing any long-lived credentials. In the **IAM & Admin** menu, open the **Service Accounts** tab and click the service account you created in Step 1 to open its details. Navigate to the **Principals with access** tab and click **Grant Access**. Add our service account as a principal and grant it both the **Service Account Token Creator** and **Service Account User** roles. Reach out to your account representative for the exact service account email to add. ![Grant role to the service account](https://storage.googleapis.com/prequel_docs/images/gcp-grant-role.png) A service account key is a long-lived credential. Use this method only when your policy prevents impersonation, and store the key securely. In the **Service accounts** list, click the **Actions** menu next to the service account from Step 1 and click **Manage keys**. ![Manage service account keys](https://storage.googleapis.com/prequel_docs/images/gcp-manage-service-account-keys.png) Click **Add key** and then **Create new key**. ![Create new key](https://storage.googleapis.com/prequel_docs/images/gcp-create-new-key.png) Select the **JSON** key type and click **Create**. Store the downloaded key securely, as you will provide its contents in Step 5. ![Create JSON key](https://storage.googleapis.com/prequel_docs/images/gcp-create-json-key.png) ## Step 4: find your project ID and data location In the Google Cloud console, open the projects list dropdown and make a note of your BigQuery **Project ID**. ![GCP Project ID](https://storage.googleapis.com/prequel_docs/images/gcp-project-id.png) Open your source dataset in BigQuery and make a note of its **Data location**, such as `us` or `us-central1`. You will provide this value as the region in Step 5. This connection supports Google Cloud organization policies that restrict identities by domain. If your organization enforces domain-restricted sharing, you can add our principal to your allow list following Google's guidance on [restricting identities by domain](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-domains). Contact your account representative for the customer ID to add. ## Step 5: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **Project ID** from Step 4. 3. The **region** (the data location) from Step 4. 4. The authentication credentials from Step 3: * If using **impersonation**: the **service account email** from Step 1. * If using a **service account key**: the **service account email** from Step 1 and the **JSON key** you downloaded. # Google cloud storage Source: https://docs.prequel.co/import/sources/google-cloud-storage Connect your Google Cloud Storage bucket as a source By default, GCS authentication uses role-based access. You create a service account in your own GCP project that holds the read permissions described below. Our service account is then granted permission to impersonate yours using short-lived tokens. No private key is shared, and you can revoke access at any time through your IAM settings. ## Step 1: create a service account In the GCP console, in the same project as your bucket, navigate to the **IAM & Admin** menu, open the **Service Accounts** tab, and click **Create service account**. ![Create service account menu](https://storage.googleapis.com/prequel_docs/images/gcp-create-service-account-menu.png) Give the service account a descriptive name and click **Create and Continue**. Click **Continue** and **Done** through the remaining optional steps without assigning any roles. Open the service account in the list and make a note of its **email**, which looks like `service-account-name@project-id.iam.gserviceaccount.com`. You will provide this email in Step 5. ## Step 2: grant read access to the bucket Navigate to the **Cloud Storage** page, click the bucket you want to read from, select the **Permissions** tab, and click **Grant access**. Add the service account you created in Step 1 as a principal and assign both the **Storage Legacy Bucket Reader** and **Storage Legacy Object Reader** roles, then click **Save**. Together these grant the `storage.buckets.get`, `storage.objects.list`, and `storage.objects.get` permissions used to read your data. ## Step 3: authorize authentication Choose the authentication method that fits your security policy. In the **IAM & Admin** menu, open the **Service Accounts** tab and click the service account you created in Step 1 to open its details. Navigate to the **Principals with access** tab and click **Grant Access**. Add our service account as a principal and grant it both the **Service Account Token Creator** and **Service Account User** roles. Reach out to your account representative for the exact service account email to add. ![Grant role to the service account](https://storage.googleapis.com/prequel_docs/images/gcp-grant-role.png) An HMAC key is a credential associated with a service account that can be used to access Google Cloud Storage. Navigate to the **Cloud Storage** page, click into the **Settings** tab, and navigate to the **Interoperability** tab. Click **Create a key for a Service Account**. ![GCS interoperability settings](https://storage.googleapis.com/prequel_docs/images/gcs-bucket-settings.png) Select the service account you created in Step 1 and click **Create key**. ![Select service account for HMAC key](https://storage.googleapis.com/prequel_docs/images/gcs-select-service-account-hmac.png) Make a note of the **Access key** and the **Secret**. You will provide these values in Step 5. A service account key is a long-lived credential. Use this method only when your policy prevents impersonation, and store the key securely. In the **Service accounts** list, click the **Actions** menu next to the service account from Step 1 and click **Manage keys**. Click **Add key**, then **Create new key**. Select the **JSON** key type and click **Create**. Store the downloaded key securely, as you will provide its contents in Step 5. ![Create JSON key](https://storage.googleapis.com/prequel_docs/images/gcp-create-json-key.png) ## Step 4: find your bucket region Navigate to the **Cloud Storage** page, click the bucket you want to read from, and make a note of its **Location**, such as `us` or `us-central1`. You will provide this value as the region in Step 5. ## Step 5: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **bucket name** of the bucket you want to read from. 3. The **region** (the bucket location) from Step 4. 4. The authentication credentials from Step 3: * If using **impersonation**: the **service account email** from Step 1. * If using **HMAC keys**: the **access key** and the **secret**. * If using a **service account key**: the **service account email** from Step 1 and the **JSON key** you downloaded. # Overview Source: https://docs.prequel.co/import/sources/overview Vendor-specific setup guides for all supported source types. Select your source vendor below for connection setup instructions. For an overview of how sources work in Prequel, see [Sources](/import/core-concepts/sources). ## Supported sources You can find all currently supported sources and documentation links below. You also may decide to self-host the source configuration instructions on your own documentation site. If you prefer to do that, we maintain a copy of the source `markdown` files accessible at public locations below: | Vendor | Type | Status | Docs | Markdown (`.md` file) | | :-------------------- | :------------- | :---------- | :------------------------------------------- | :----------------------------------------------------------------------------------------------- | | `snowflake` | OLAP | Available | [link](/import/sources/snowflake) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-snowflake.md) | | `bigquery` | OLAP | Available | [link](/import/sources/bigquery) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-bigquery.md) | | `redshift` | OLAP | Available | [link](/import/sources/redshift) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-redshift.md) | | `redshift_serverless` | OLAP | Available | [link](/import/sources/redshift-serverless) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-redshift_serverless.md) | | `databricks` | OLAP | Coming soon | | | | `athena` | OLAP | Coming soon | | | | `clickhouse` | OLAP | Coming soon | | | | `motherduck` | OLAP | Coming soon | | | | `postgres` | OLTP | Available | [link](/import/sources/postgres) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-postgres.md) | | `mysql` | OLTP | Coming soon | | | | `aurora_mysql` | OLTP | Coming soon | | | | `sql_server` | OLTP | Coming soon | | | | `oracle` | OLTP | Coming soon | | | | `singlestore` | OLTP | Coming soon | | | | `s3` | Object Storage | Available | [link](/import/sources/s3) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-s3.md) | | `s3_compatible` | Object Storage | Available | [link](/import/sources/s3-compatible) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-s3_compatible.md) | | `gcs` | Object Storage | Available | [link](/import/sources/google-cloud-storage) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-gcs.md) | | `abs` | Object Storage | Available | [link](/import/sources/azure-blob-storage) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-abs.md) | | `sftp` | Object Storage | Available | [link](/import/sources/sftp) | [link](https://storage.googleapis.com/prequel_docs/documentation/sources-sftp.md) | # PostgreSQL Source: https://docs.prequel.co/import/sources/postgres Connect your PostgreSQL database as a source To avoid putting unnecessary load on your primary database during data transfer, we recommend connecting to a read replica. Step 1 covers creating one on AWS. This step is optional, and you can connect directly to your primary instance if you prefer. ## Step 1: create a read replica (optional) In your Amazon RDS dashboard, click the PostgreSQL instance you want to replicate. On the database page, click **Actions**, then select **Create read replica** from the drop-down. ![Create read replica](https://storage.googleapis.com/prequel_docs/images/rds-create-read-replica.png) In the **Instance specifications** section, specify the instance type for the read replica. It can be smaller than the main instance. ![Read replica instance class](https://storage.googleapis.com/prequel_docs/images/rds-instance-class-small.png) In the **Network & Security** section, under **Publicly accessible**, select **Yes** so that the read replica is reachable from outside your VPC. It remains accessible only through whitelisted IPs. If you plan to connect through an SSH tunnel, you can leave this set to **No**. ![Publicly accessible setting](https://storage.googleapis.com/prequel_docs/images/rds-publicly-accessible-yes.png) In the **Settings** section, enter a **DB instance identifier**, such as `source-read-replica`, then click **Create read replica**. ![Read replica identifier](https://storage.googleapis.com/prequel_docs/images/rds-read-replica-identifier.png) In your Amazon RDS dashboard, click the Aurora PostgreSQL instance you want to add a reader to. On the database page, click **Actions**, then select **Add reader** from the drop-down. ![Add reader](https://storage.googleapis.com/prequel_docs/images/aurora-add-reader.png) In the **Settings** section, enter a **DB instance identifier**, such as `source-reader`. In the **DB instance class** section, specify the instance type. It can be smaller than the main instance, though AWS may set a lower bound. In the **Connectivity** section, select **Publicly accessible** so that the reader is reachable from outside your VPC. It remains accessible only through whitelisted IPs. If you plan to connect through an SSH tunnel, you can leave this set to **Not publicly accessible**. Click **Add reader**. ![Publicly accessible setting](https://storage.googleapis.com/prequel_docs/images/postgres-publicly-accessible.png) ## Step 2: allow network access Allow read access to your PostgreSQL database, or to the read replica you created in Step 1, from the static IP. Reach out to your account representative for the static IP address to use. ### Configure the security group In your **Amazon RDS** > **Databases** list, click the PostgreSQL instance you want to connect. In the **Connectivity & security** tab, make a note of the **Endpoint** and the **Port** number. ![Instance endpoint and port](https://storage.googleapis.com/prequel_docs/images/aurora-reader-access.png) Click one of the VPC security groups (usually `default`). VPC groups are permissive rather than restrictive, so for instances with multiple security groups, only one needs the new inbound rule. ![VPC security groups](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-security-groups.png) Select the **Inbound rules** tab, click **Edit inbound rules**, then click **Add rule**. Set the rule type to **Custom TCP**, set the **Port range** to the port you noted (usually `5432`), and set a `Custom` **Source** value that includes the static IP. You need to add `/32` to the end of the IP to express it in CIDR notation. Click **Save rules**. ![Add inbound rule](https://storage.googleapis.com/prequel_docs/images/postgres-add-rule.png) ### Configure the network ACL For database instances in a VPC, you also need to allow traffic at the network ACL level. In your RDS dashboard, select the PostgreSQL instance, then click the link to the instance's VPC. ![Instance VPC](https://storage.googleapis.com/prequel_docs/images/aurora-reader-vpc.png) Click the **VPC ID**. In the **Details** section, click the link under **Main network ACL**, then click the network ACL ID. ![VPC ID](https://storage.googleapis.com/prequel_docs/images/postgres-vpc-id.png) ![Network ACL ID](https://storage.googleapis.com/prequel_docs/images/postgres-network-acl-id.png) Click the **Inbound rules** tab and check for an existing rule with a **Source** of `0.0.0.0/0` set to `Allow`. This is a default rule created by AWS, and if it already exists, you can skip to the outbound rules. Otherwise, click **Edit inbound rules**, add a rule allowing access to your database port (usually `5432`) from the static IP, and click **Save changes**. ![Inbound rules](https://storage.googleapis.com/prequel_docs/images/postgres-inbound-rules.png) Select the **Outbound rules** tab and check for an existing rule with a **Destination** of `0.0.0.0/0` set to `Allow`. This is a default rule created by AWS, and if it already exists, you are done. Otherwise, click **Edit outbound rules** and add a rule allowing outbound traffic to ports `1024` to `65535` for **Destination** `0.0.0.0/0`. ![Outbound rules](https://storage.googleapis.com/prequel_docs/images/postgres-outbound-rules.png) In your firewall or security group, create a rule that allows: * Incoming connections to your host and port (usually `5432`) from the static IP. * Outgoing connections from ports `1024` to `65535` to the static IP. If your database is not reachable over the public internet, you can connect through an SSH tunnel instead. Provide the bastion host details in Step 4, and add the public key from your account representative to the bastion's `authorized_keys` file. ## Step 3: create a read-only user Open a connection to your PostgreSQL database using a SQL client. Create a dedicated user by running the following command. Replace `` and `` with values of your choice. ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD '' NOSUPERUSER NOCREATEDB NOCREATEROLE; ``` Grant the user read-only access to the specific tables you want to sync. Replace `` with the schema that contains those tables, and grant `SELECT` on each table individually. ```sql title="Grant table access" icon="database" expandable theme={null} GRANT USAGE ON SCHEMA "" TO ; GRANT SELECT ON ""."" TO ; GRANT SELECT ON ""."" TO ; ``` Repeat this for every schema that contains tables you want to sync. To grant access to every table in a schema instead of listing tables individually, grant `SELECT` on all tables in the schema. ```sql title="Grant all tables" icon="database" theme={null} GRANT USAGE ON SCHEMA "" TO ; GRANT SELECT ON ALL TABLES IN SCHEMA "" TO ; ``` ## Step 4: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **host** (for example, `your-db.sd8jekhrlkhla.us-east-1.rds.amazonaws.com`). 3. The **port** (most likely `5432`). 4. The **database** you want to read from. 5. The **schema** from Step 3. 6. The **username** from Step 3. 7. The **password** from Step 3. The connection uses SSL by default. If you are connecting through an SSH tunnel, also provide the **SSH host**, **SSH port**, and **SSH username** for your bastion server. # Redshift Source: https://docs.prequel.co/import/sources/redshift Connect your Redshift data warehouse as a source ## Step 1: find cluster connection details 1. In the Redshift console, click **Clusters**. 2. Select the cluster you would like to connect. ![Redshift cluster](https://storage.googleapis.com/prequel_docs/images/redshift-cluster.png) In the **General information** pane, make note of the **Endpoint** details, the **Cluster identifier**, and the **AWS region** that the cluster is hosted in. You may need to use the **copy** icon to copy the full details to discover the full endpoint and port number. ![Redshift endpoint details](https://storage.googleapis.com/prequel_docs/images/redshift-endpoint-details.png) ## Step 2: whitelist connection 1. In the Redshift console, click **Clusters**. 2. Select the cluster you would like to connect. 3. Click the **Properties** tab. 4. Scroll down to the **Network and security settings** section. 5. In the VPC security group field, select a security group to open it. ![Redshift VPC security groups](https://storage.googleapis.com/prequel_docs/images/redshift-vpc-security-groups.png) 1. In the Security Groups window, click **Inbound rules**. 2. Click **Edit inbound rules**. 3. In the Edit the Inbound rules window, create a custom TCP rule for the static IP: 1. Select **Custom TCP** in the drop-down menu. 2. Enter your Redshift port number (likely `5439`). 3. Enter the static IP address. 4. Click **Add rule**. ## Step 3: create a limited user Connect to Redshift using the SQL client. Execute the following query to create a user (replace `` with a password of your choice). ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD ''; ``` Execute the following query to grant the user read-only access to the specific tables you want to sync (replace `` with your schema name): ```sql title="Grant table access" icon="database" expandable theme={null} GRANT USAGE ON SCHEMA TO ; GRANT SELECT ON . TO ; GRANT SELECT ON . TO ; ``` To grant access to every table in a schema instead of listing tables individually, grant `SELECT` on all tables in the schema. ```sql title="Grant all tables" icon="database" theme={null} GRANT USAGE ON SCHEMA TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ``` ## Step 4: configure AWS authentication You must also provide AWS credentials for cluster access. You can authenticate with either an **IAM role** (recommended) or **AWS access keys**. In the AWS IAM console, create a new policy with the JSON below. Replace `REGION_NAME`, `ACCOUNT_ID`, `CLUSTER_NAME`, `USERNAME` (the user from Step 3), and `DATABASE_NAME` with values that match your cluster. ```json title="IAM policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "redshift:GetClusterCredentials", "Resource": [ "arn:aws:redshift:REGION_NAME:ACCOUNT_ID:dbuser:CLUSTER_NAME/USERNAME", "arn:aws:redshift:REGION_NAME:ACCOUNT_ID:dbname:CLUSTER_NAME/DATABASE_NAME" ] } ] } ``` In the AWS IAM console, create a new role using the custom trust policy below, and attach the permissions policy you created in the previous step. Reach out to your contact for the value of ``. ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` After the role is created, copy its **ARN** (for example, `arn:aws:iam::123456789012:role/source-redshift`). You will provide this ARN in Step 5. In the AWS IAM console, create a new user with programmatic access to the Redshift cluster you configured in Step 1. Generate an access key pair for the user. Make note of the **access key ID** and the **secret access key**. You will provide these values in Step 5. ## Step 5: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source 2. The **host** (e.g., `source-redshift.cw1drqwgbdue.us-east-1.redshift.amazonaws.com`) 3. The **port** \[e.g., `5439`] 4. The **database** for your Redshift cluster. 5. The **cluster identifier** from Step 1. 6. The **AWS region** from Step 1. 7. The **username** from Step 3. 8. The authentication credentials from Step 4: * If using an **IAM role**: the **IAM role ARN** * If using **AWS access keys**: the **password** from Step 3, the **AWS access key ID**, and the **AWS secret access key** # Redshift serverless Source: https://docs.prequel.co/import/sources/redshift-serverless Connect your Redshift Serverless workgroup as a source Redshift Serverless issues temporary database credentials per workgroup, so the connection involves two identities: * A **Redshift database user** in your workgroup. You create this read-only user in Step 3. It owns the schema and table privileges needed to read your data. * An **AWS IAM role or access keys** in your AWS account. You configure this in Step 5. It holds the permission to call `redshift-serverless:GetCredentials` on your workgroup and the S3 permissions used to stage data during a sync. We use this identity to obtain short-lived database credentials, so no long-lived database password is required. ## Step 1: find workgroup connection details 1. In the Redshift console, click **Workgroups**. 2. Select the workgroup you would like to connect, and make note of the **workgroup** name. In the **General information** pane, make note of the **Endpoint** details and the **AWS region** that the workgroup is hosted in. You may need to use the **copy** icon to copy the full details to discover the full endpoint and port number. ![Redshift endpoint details](https://storage.googleapis.com/prequel_docs/images/redshift-endpoint-details.png) ## Step 2: whitelist connection 1. In the Redshift console, click **Workgroups**. 2. Select the workgroup you would like to connect. 3. Click the **Properties** tab. 4. Scroll down to the **Network and security settings** section. 5. In the VPC security group field, select a security group to open it. ![Redshift VPC security groups](https://storage.googleapis.com/prequel_docs/images/redshift-vpc-security-groups.png) 1. In the Security Groups window, click **Inbound rules**. 2. Click **Edit inbound rules**. 3. In the Edit the Inbound rules window, create a custom TCP rule for the static IP: 1. Select **Custom TCP** in the drop-down menu. 2. Enter your Redshift port number (likely `5439`). 3. Enter the static IP address. 4. Click **Add rule**. ## Step 3: create a limited user Connect to your Redshift Serverless workgroup using the SQL client. Execute the following query to create a user. Because authentication uses temporary credentials issued by `redshift-serverless:GetCredentials`, a password is not required and the user can be created with `PASSWORD DISABLE`. ```sql title="Create user" icon="database" theme={null} CREATE USER PASSWORD DISABLE; ``` Execute the following query to grant the user read-only privileges (replace `` with your schema name): ```sql title="Grant read-only access" icon="database" theme={null} GRANT USAGE ON SCHEMA TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ``` ## Step 4: create a staging bucket Redshift Serverless reads use the high-throughput `UNLOAD` path, which stages data in an S3 bucket in your account before it is read. Files are cleaned up automatically after each sync. 1. Navigate to the S3 service page. 2. Click **Create bucket**. 3. Enter a **Bucket name** and modify any of the default settings as desired. **Object Ownership** can be set to **ACLs disabled** and **Block Public Access settings for this bucket** can be set to **Block all public access** as recommended by AWS. Make note of the bucket name and AWS region. 4. Click **Create bucket**. You may configure a lifecycle rule on the staging bucket to automatically delete objects older than two days, as the bucket is not used to persist data. In the bucket **Management** tab, click **Create lifecycle rule** and set an expiration action for current versions of objects with a two-day age. Sync logic cleans up files after each sync completes, so this is an optional step. ## Step 5: configure AWS authentication You must provide AWS credentials for workgroup access. You can authenticate with either an **IAM role** (recommended) or **AWS access keys**. In both cases, the identity needs permission to call `redshift-serverless:GetCredentials` on your workgroup and to read, write, and delete objects in the staging bucket from Step 4. In the AWS IAM console, create a new policy with the JSON below. Replace `REGION_NAME`, `ACCOUNT_ID`, and `WORKGROUP_NAME_OR_ID` with values that match your workgroup, and replace `BUCKET_NAME` with the staging bucket from Step 4. The first bucket permission applies to `BUCKET_NAME`, whereas the second applies only to the bucket's contents at `BUCKET_NAME/*`. This is an important distinction. ```json title="IAM policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "redshift-serverless:GetCredentials" ], "Resource": [ "arn:aws:redshift-serverless:REGION_NAME:ACCOUNT_ID:workgroup/WORKGROUP_NAME_OR_ID" ] }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" }, { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::BUCKET_NAME/*" } ] } ``` In the AWS IAM console, create a new role using the custom trust policy below, and attach the permissions policy you created in the previous step. Reach out to your contact for the value of ``. ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` After the role is created, copy its **ARN** (for example, `arn:aws:iam::123456789012:role/source-redshift-serverless`). You will provide this ARN in Step 6. In the AWS IAM console, create a new policy with the same JSON shown in the IAM role tab, granting `redshift-serverless:GetCredentials` on your workgroup and the S3 permissions on your staging bucket. In the AWS IAM console, create a new user with programmatic access, and attach the policy you created in the previous step. Generate an access key pair for the user. Make note of the **access key ID** and the **secret access key**. You will provide these values in Step 6. ## Step 6: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **workgroup** name from Step 1. 3. The **host** (e.g., `workgroup.123456789.us-east-1.redshift-serverless.amazonaws.com`). 4. The **port** \[e.g., `5439`]. 5. The **database** for your Redshift Serverless workgroup. 6. The **username** from Step 3. 7. The **S3 bucket name** and **S3 bucket region** from Step 4. 8. The authentication credentials from Step 5: * If using an **IAM role**: the **IAM role ARN**. * If using **AWS access keys**: the **AWS access key ID** and the **AWS secret access key**. # Amazon S3 Source: https://docs.prequel.co/import/sources/s3 Connect your S3 bucket as a source By default, S3 authentication uses role-based access. You create an IAM role in your own AWS account that holds the read permissions described below, and our identity is granted permission to assume it. No long-lived credentials are shared, and you can revoke access at any time through your IAM settings. ## Step 1: configure AWS authentication Choose the authentication method that fits your security policy. In the AWS IAM console, navigate to the **Policies** tab and click **Create policy**. Click the **JSON** tab and paste the following policy, replacing `BUCKET_NAME` with the name of the bucket you want to read from. ```json title="IAM policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::BUCKET_NAME/*" }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET_NAME" } ] } ``` If your S3 bucket uses KMS encryption (CMK), add the following statement to the `Statement` array to allow decryption with your KMS key. Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values. ```json title="KMS decrypt statement" icon="brackets-curly" theme={null} { "Effect": "Allow", "Action": "kms:Decrypt", "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID" } ``` Name the policy, add a description, and click **Create policy**. In the AWS IAM console, navigate to the **Roles** tab and click **Create role**. Select **Custom trust policy** and paste the trust policy below to allow our identity to assume the role. Reach out to your account representative for the value of ``. ```json title="Trust policy" icon="brackets-curly" expandable theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "sts:AssumeRoleWithWebIdentity" ], "Principal": { "Federated": "accounts.google.com" }, "Condition": { "StringEquals": { "accounts.google.com:sub": "" } } } ] } ``` Click **Next**, attach the permissions policy you created in the previous step, enter a **Role name**, for example, `source-s3`, and click **Create role**. Once the role is created, search for it in the Roles list, click the role name, and make a note of its **ARN** (for example, `arn:aws:iam::123456789012:role/source-s3`). You will provide this ARN in Step 2. In the AWS IAM console, navigate to the **Users** tab and click **Add users**. Enter a **User name** for the service, for example, `source-service`, and select programmatic access. Attach the read policy described in the IAM role tab (`s3:GetObject` and `s3:ListBucket` on the target bucket) directly to the user. Generate an access key pair for the user and make a note of the **access key ID** and the **secret access key**. You will provide these values in Step 2. If your bucket policy restricts access by IP, allow the static IP so we can reach the bucket. Reach out to your account representative for the static IP address to use. ## Step 2: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **bucket name** of the bucket you want to read from. 3. The **AWS region** the bucket is hosted in. 4. The authentication credentials from Step 1: * If using an **IAM role**: the **role ARN**. * If using **AWS access keys**: the **access key ID** and the **secret access key**. # S3-compatible storage Source: https://docs.prequel.co/import/sources/s3-compatible Connect your S3-compatible object store as a source Many object storage platforms offer "S3 compatibility", enabling reading from the object store using the S3 protocol. The S3 protocol uses an HMAC key made up of an access ID and a secret to authenticate and read data. ## Step 1: create an HMAC access ID and secret Consult your object storage platform's documentation to learn how to generate an HMAC access ID and secret. Scope the credentials to read only the bucket you want to sync. If your object store restricts access by IP, allow the static IP so we can reach the bucket. Reach out to your account representative for the static IP address to use. ## Step 2: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **host** of your S3-compatible server. 3. The **port** of your S3-compatible server. 4. The **bucket name** you want to read from. 5. The **HMAC access ID** from Step 1. 6. The **HMAC secret** from Step 1. The connection uses SSL by default. If your server does not support SSL, you can disable it. # SFTP Source: https://docs.prequel.co/import/sources/sftp Connect your SFTP server as a source By default, SFTP uses keypair authentication for access. We generate the keypair and provide you with the public key to register on a user on your SFTP server. The private key is securely generated and stored in our system and is never shared externally. ## Step 1: locate your public key Locate the public key provided to you. The public key will look roughly like this: ```shell title="Public key" icon="terminal" theme={null} ssh-rsa some-comment ``` You will register this public key on the SFTP user you create in Step 2. Reach out to your account representative if you have not been provided a public key. ## Step 2: create a user on the SFTP server Log in to the SFTP server and complete the steps below. Create the `sftpreader` group. ```shell title="Create the group" icon="terminal" theme={null} sudo groupadd sftpreader ``` Create the `sftpreader` user. ```shell title="Create the user" icon="terminal" theme={null} sudo useradd -m -g sftpreader sftpreader ``` Switch to the `sftpreader` user, create the `.ssh` directory, and set its permissions. ```shell title="Create the .ssh directory" icon="terminal" theme={null} sudo su - sftpreader mkdir ~/.ssh chmod 700 ~/.ssh ``` Create the `authorized_keys` file and set its permissions. ```shell title="Create authorized_keys" icon="terminal" theme={null} cd ~/.ssh touch authorized_keys chmod 600 authorized_keys ``` Add the public key from Step 1 to the `authorized_keys` file. The key, including the `ssh-rsa` prefix and comment, should be all on one line in the file, without linebreaks. ```shell title="Add the public key" icon="terminal" theme={null} echo "ssh-rsa sftpreader-public-key" >> authorized_keys ``` Ensure the `sftpreader` user has read access to the directory that contains the files you want to sync. The SFTP protocol encrypts files in transit. Allow the static IP to connect on your SFTP port (usually `22`) at your firewall or network perimeter. Reach out to your account representative for the static IP address to use. ## Step 3: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source. 2. The **host** of your SFTP server. 3. The **port** (most likely `22`). 4. The **username** from Step 2. # Snowflake Source: https://docs.prequel.co/import/sources/snowflake Connect your Snowflake data warehouse as a source ## Step 1: locate your public key Locate the public key provided to you. The public key will be a long string of text, loosely resembling the format: `MIIBI......Xrw2nwIDAQAB`. You will register this public key on the Snowflake user you create in Step 2. ## Step 2: create role, user, and warehouse To complete the following setup steps, you or a Snowflake admin on your team must have the `SECURITYADMIN` and `SYSADMIN` roles. To check your account for these roles, run `SHOW GRANTS TO USER ;` and review the `role` column. Review and make any changes to the following setup script. Update `database_name` and `schema_name` to point at the database and schema you want to read from, list the tables you want to sync in the grant statements, and paste the public key from Step 1 into `RSA_PUBLIC_KEY`. ```sql title="Setup script" icon="database" expandable theme={null} begin; -- create variables for user / role / warehouse set user_name = 'TRANSFER_USER'; -- all letters must be uppercase set role_name = 'TRANSFER_ROLE'; -- all letters must be uppercase set warehouse_name = 'TRANSFER_WAREHOUSE'; -- all letters must be uppercase set database_name = 'SOURCE_DATABASE'; -- all letters must be uppercase set schema_name = 'SOURCE_SCHEMA'; -- all letters must be uppercase -- change role to securityadmin for user / role steps use role securityadmin; -- create role for the transfer service create role if not exists identifier($role_name); grant role identifier($role_name) to role SYSADMIN; -- establish SYSADMIN as the parent of the new role. Note: this does not grant the access privileges of SYSADMIN to the new role. -- create a user for the transfer service create user if not exists identifier($user_name) RSA_PUBLIC_KEY='MIIBIjANBgkqh...'; -- replace with the complete public key from Step 1 -- set default role and warehouse, and mark the user as a service account alter user identifier($user_name) SET default_role = $role_name; alter user identifier($user_name) SET default_warehouse = $warehouse_name; alter user identifier($user_name) SET type = service; grant role identifier($role_name) to user identifier($user_name); -- change role to sysadmin for warehouse / database steps use role sysadmin; -- create a warehouse for the transfer service create warehouse if not exists identifier($warehouse_name) warehouse_size = xsmall warehouse_type = standard auto_suspend = 60 auto_resume = true initially_suspended = true; -- grant warehouse and database access to the transfer role grant USAGE on warehouse identifier($warehouse_name) to role identifier($role_name); grant USAGE on database identifier($database_name) to role identifier($role_name); -- grant schema and table read access to the transfer role use database identifier($database_name); grant USAGE on schema identifier($schema_name) to role identifier($role_name); grant SELECT on table identifier($schema_name || '.') to role identifier($role_name); -- repeat for each table you want to sync grant SELECT on table identifier($schema_name || '.') to role identifier($role_name); commit; ``` By default, this script creates a new warehouse. If you would prefer to use an existing warehouse, change the `warehouse_name` variable from `TRANSFER_WAREHOUSE` to the name of the warehouse to be shared. To grant access to every table in the schema instead of listing tables individually, replace the per-table grants with a single grant on all tables in the schema. ```sql title="Grant all tables" icon="database" theme={null} grant SELECT on all tables in schema identifier($schema_name) to role identifier($role_name); ``` In the Snowflake interface, select the dropdown next to the **Run** button, and click **Run All**. This will run every query in the script at once. If successful, you will see `Statement executed successfully` in the query results. ## Step 3: configure the Snowflake access policy If your Snowflake data warehouse is using Snowflake Access Policies, a new policy must be added to allow the static IP to read from the warehouse. Reach out to your contact for the static IP address to use. Run the following to check for existing IP safelists. ```sql title="Show network policies" icon="database" theme={null} SHOW NETWORK POLICIES; ``` If there is no existing Snowflake network policy (the `SHOW` query returns no results), you can skip to Step 4. If there is an existing Snowflake network policy, you must alter the existing policy or create a new one to safelist the static IP address. Run the following to create a new network policy that safelists the static IP address. ```sql title="Create network policy" icon="database" theme={null} CREATE NETWORK POLICY ALLOWED_IP_LIST = (''); ``` If you have no existing network policies and you create your first as part of this step, all other IPs outside of the `ALLOWED_IP_LIST` will be blocked. Snowflake does not allow setting a network policy that blocks your current IP address, and will return an error if you try. Be careful when setting your first network policy. ## Step 4: submit your connection details Provide the following details to complete the source setup: 1. The **name** is a descriptive name of the source 2. The **host** (e.g., `account.us-central1.gcp.snowflakecomputing.com`) 3. The **port** \[e.g., `443`] 4. The **database** from Step 2 5. The **username** from Step 2