Usage data

Overview

You can receive your Prequel usage data in your own data warehouse, using any of our supported destination vendors. Contact Prequel support to enable and configure this feature. Once you configure your usage data destination, Prequel syncs three tables to your destination: audit_log, destinations_snapshot, and model_transfers_log. By default, this data is refreshed hourly.

Data format

Prequel syncs three tables to your destination: audit_log, destinations_snapshot, and model_transfers_log.

`audit_log`

Each row in this table corresponds to one event in the audit log, and is the same data available via the /logs endpoint. These logs are intended to be a comprehensive record of all write actions taken on your Prequel account. For more information, including the events included in this table, see the audit log documentation.

Field Name	Data Type	Description
`event_id`	`string`	Audit log ID. This ID is the same as the `/logs` endpoint.
`occurred_at`	`timestamp`	Timestamp when event occurred, corresponds to `created_at` in logs endpoint, reported in UTC.
`event_type`	`string`	The type of event.
`resource_id`	`string`	The ID of the resource involved in the event.
`actor_id`	`string`	The identifier of the actor who initialized the event. Some historical events have the value `unknown`.
`actor_type`	`string`	The type of actor who initialized the event. The possible values are `unknown`, `user`, `api_key`, `recipient`.
`current_resource`	`json`	Nullable JSON object representing the current state of the resource. DELETE operations will never have this field populated. Some historical events do not have this field populated.
`previous_resource`	`json`	Nullable JSON object representing the previous state of the resource. CREATE operations will never have this field populated. Some historical events do not have this field populated.

`destinations_snapshot`

Each entry in this table corresponds to one destination in your Prequel account. Note that this table represents the current state of destinations configured in Prequel. To get a historical record of actions taken on destinations, you can use the audit log data.

Field Name	Data Type	Description
`destination_id`	`string`	Destination ID in the Prequel API.
`is_enabled`	`bool`	Whether scheduled syncs are enabled on this destination.
`updated_at`	`timestamp`	Timestamp when the destination was last updated.
`created_at`	`timestamp`	Timestamp corresponding to when the destination was first created.
`recipient_id`	`string`	The ID of the recipient the destination is assigned to.
`id_in_provider_system`	`string`	The ID in the source provider system, corresponds to the tenant schema in schema tenanting mode.
`products`	`json`	JSON array of objects representing assigned products. Each entry has the fields `id` and `name`.
`frequency_minutes`	`int`	The frequency at which Prequel’s scheduler enqueues transfers to this destination.
`enabled_models`	`json`	JSON array representing the enabled models for this destination. Each entry has the fields `id` and `name`.
`has_all_models_enabled`	`bool`	Indicates whether the destination inherits all current and future models on its products.
`destination_name`	`string`	The name of the destination.
`destination_vendor`	`string`	The vendor of the destination.
`max_concurrent_transfers`	`int`	Maximum number of simultaneous transfers allowed.
`max_concurrent_queries_per_transfer`	`int`	Maximum number of concurrent queries allowed per transfer.
`deleted_at`	`timestamp`	Timestamp when the destination was deleted. Null for active destinations.

`model_transfers_log`

Each entry in this table corresponds to one completed model transfer within a transfer job. Model transfers that are pending or in progress are not currently included in this table.

Field Name	Data Type	Description
`model_transfer_id`	`string`	Model ID concatenated with transfer ID. This ID only appears within Prequel’s usage data.
`transfer_id`	`string`	The ID of the transfer in the API.
`model_id`	`string`	The ID of the model being transferred.
`model_transfer_id`	`string`	A unique ID for the model transfer, combining transfer ID and model ID.
`model_name`	`string`	The name of the model being transferred.
`source_id`	`string`	The ID of the source from which the model was read.
`source_name`	`string`	The name of the source from which the model was read.
`source_vendor`	`string`	The vendor of the source from which the model was read.
`destination_id`	`string`	The ID of the destination to which the model was written.
`recipient_id`	`string`	The ID of the recipient for the destination.
`id_in_provider_system`	`string`	The ID in the provider system, corresponds to the tenant schema.
`destination_name`	`string`	The name of the destination.
`destination_vendor`	`string`	The vendor type of the destination.
`is_full_refresh`	`bool`	Indicates whether the transfer was a full refresh.
`transfer_submitted_at`	`timestamp`	Timestamp when the transfer was submitted.
`transfer_started_at`	`timestamp`	Timestamp when the worker began executing the model transfer.
`transfer_ended_at`	`timestamp`	Timestamp when the model transfer completed.
`transfer_start_window_at`	`timestamp`	Lower bound timestamp for the data transferred.
`transfer_end_window_at`	`timestamp`	Upper bound timestamp for the data transferred.
`status`	`string`	The status of the model transfer (e.g., `SUCCESS`, `ERROR`, `CANCELLED`).
`enqueueing_actor_id`	`string`	The identifier of the actor who enqueued the transfer.
`enqueueing_actor_type`	`string`	The type of actor who enqueued the transfer, taking one of the following values: `user`, `api_key`, `service`, or `unknown`.
`error_trace`	`string`	Detailed error message in case of a transfer error.
`error_code`	`string`	Error code corresponding to the transfer error.
`error_blame`	`string`	The party responsible for the error.
`error_message`	`string`	Error message for the model transfer.
`documentation_url`	`string`	URL to documentation for further information on the error.
`transfer_log`	`json`	Transfer log information.
`billing_date`	`date`	The date on which Prequel will bill for this transfer. This is the date of the transfer `ended_at` at time zone America/Los_Angeles.
`rows_transferred`	`bigint`	Total number of rows selected from the source for this model during this transfer.
`delta_rows_transferred`	`bigint`	Estimated number of rows in this transfer with a `last_modified_at` newer than the latest previously completed transfer for this model (does not include lookback overlap).
`volume_transferred_in_mb`	`float`	The memory footprint of the data transferred.

WarningDeleting a model can cause historical transfer records to lose their associated model and source details.This issue is fixed for future transfers, but the time period of affected historical data varies by Prequel instance. Please contact support for more information.Transfers for deleted destinations may not be available in model_transfers_log prior to a certain date, depending on your deployment. For more details about transfers for deleted destinations, please contact support.

Staging usage data

Both production and staging environment data from Prequel is synced. Staging environment data is synced as separate tables to the same destination with a _staging suffix:

audit_log_staging
destinations_snapshot_staging
model_transfers_log_staging

Example queries

How many rows have we transferred over the past month for this destination, grouped by day?

Rows transferred per day

SELECT
  DATE(transfer_ended_at) AS transfer_date,
  SUM(rows_transferred) AS total_rows_transferred
FROM
  `{{your-schema}}.model_transfers_log`
WHERE
  destination_id = '{{your-destination-id}}'
  AND transfer_ended_at >= CAST(CURRENT_DATE - INTERVAL '1 MONTH' AS TIMESTAMP)
GROUP BY
  transfer_date
ORDER BY
  transfer_date;

For a given destination and model, what is the latest transfer status?

Latest transfer status

WITH LatestStatus AS (
  SELECT
    model_id,
    MAX(transfer_ended_at) AS last_transfer_time,
    ARRAY_AGG(status ORDER BY transfer_ended_at DESC LIMIT 1)[OFFSET(0)] AS latest_status
    MAX(CASE WHEN is_full_refresh THEN transfer_ended_at ELSE NULL END) AS last_full_refresh_time
  FROM
    `{{your-schema}}.model_transfers_log`
  WHERE
    destination_id = '{{your-destination-id}}'
  GROUP BY
    model_id
)
SELECT
  l.model_id,
  l.latest_status,
  l.last_transfer_time,
  l.last_full_refresh_time,
  CASE
    -- Have we ever triggered a transfer for this model?
    WHEN l.latest_status IS NULL THEN false
    ELSE true
  END AS transfer_has_been_triggered
FROM
  LatestStatus l
LEFT JOIN
  `{{your-schema}}.destinations_snapshot` d
  ON d.destination_id = '{{your-destination-id}}';

For a given billing date and destination, what are the transfer metrics?

Transfer metrics by billing date

SELECT
  billing_date,
  destination_id,
  COUNT(*) AS count_transfers,
  COUNTIF(status = 'SUCCESS') AS count_transfers_succeeded,
  COUNTIF(status = 'ERROR') AS count_transfers_error,
  SUM(COALESCE(rows_transferred, 0)) AS total_rows_transferred,
  SUM(COALESCE(volume_transferred_in_mb, 0)) AS total_volume_transferred_mb,
  SUM(COALESCE(delta_rows_transferred, 0)) AS total_deltas_transferred,
  SUM(TIMESTAMP_DIFF(transfer_ended_at, transfer_started_at, SECOND)) AS total_transfer_time,
  SUM(TIMESTAMP_DIFF(transfer_started_at, transfer_submitted_at, SECOND)) AS total_queue_wait_time,
  SUM(TIMESTAMP_DIFF(transfer_ended_at, transfer_started_at, SECOND)) AS total_execution_time
FROM
  `{{your-schema}}.model_transfers_log`
GROUP BY
  billing_date, destination_id
ORDER BY
  billing_date DESC;

What is the update history for a given destination?

Destination update history

SELECT
  DATE(occurred_at) AS update_date,
  occurred_at AS update_timestamp,
  actor_id,
  actor_type,
  resource_id AS destination_id,
  previous_resource,
  current_resource
FROM
  `{{your-schema}}.audit_log`
WHERE
  event_type = 'DESTINATION_UPDATE'
  AND resource_id = '{{your-destination-id}}'
ORDER BY
  occurred_at DESC;

Getting started

Core concepts

Features

Deploying Prequel

Logging & Monitoring

Integrations

Developer SDKs

Sources

Destinations

Security & compliance

Overview

Data format

`audit_log`

`destinations_snapshot`

`model_transfers_log`

Staging usage data

Example queries

​Overview

​Data format

​audit_log

​destinations_snapshot

​model_transfers_log

​Staging usage data

​Example queries

Overview

Data format

`audit_log`

`destinations_snapshot`

`model_transfers_log`

Staging usage data

Example queries