Sensitive Trace Webhooks

Overview

As a matter of company policy and compliance, Prequel never stores or retains any of the data it transfers.

Prequel occasionally encounters errors during data transfers, typically due to issues with the source or destination database systems. When these rare errors occur, the trace surfaced by the relevant database driver may occasionally include data included in the transfer. Note that this is entirely outside of Prequel's control.

To uphold its data privacy and security guarantees, Prequel handles errors as follows:

1. In the Prequel application and Prequel API endpoints, Prequel surfaces a coded error message which will never contain sensitive customer data. Raw traces are explicitly redacted from these messages to guarantee that no sensitive data is surfaced.
2. Prequel provides Sensitive Trace Webhooks to enable customers to receive unmodified errors in their own infrastructure standardized for a generic HTTP POST endpoint, providing additional context for debugging errors while keeping sensitive data out of standard logs.

⚠️

Important: These payloads may include sensitive information. Ensure your receiver is authenticated, access-controlled, and monitored according to your policies.

When are sensitive traces sent?

• Error details are sent per-transfer, aggregating errors across models in an array of traces. These are delivered as a single POST request per endpoint configured.
• If no sensitive traces occur during a transfer, no POST request is sent.
• Sensitive traces are also sent for source and destination connection tests. Sensitive traces are not forwarded for model validation errors.
• One POST request is made per-transfer to each sensitive trace endpoint configured.

Configuration steps

Step 1: Set up your receiver

Before creating the endpoint in Prequel, ensure your receiver is ready to accept POST requests. This can be any HTTPS endpoint you control -- an internal service, a Slack Incoming Webhook, a Datadog HTTP endpoint, or similar.

If you are routing sensitive traces to Slack, you can use the generic Prequel webhook guide to configure your receiver:

• Slack Integration

Step 2: Create the sensitive trace endpoint

Use the Create Sensitive Trace Endpoint API to register your receiver with Prequel.

Request:

POST /sensitive-trace-endpoints

Generic Webhook example:

{
  "sensitive_trace_endpoint": {
    "name": "Production Error Webhook",
    "description": "Sends errors to our internal monitoring system",
    "url": "https://your-domain.com/webhooks/prequel-errors",
    "secret": "your-webhook-secret-token",
    "header_template": "Authorization: Bearer {{.Secret}}\nContent-Type: application/json",
    "body_template": "{\"environment\": \"{{.Environment}}\", \"timestamp\": \"{{.Timestamp}}\", \"destination_url\": \"https://YOUR_ADMIN_HOST/destinations/{{.DestinationId}}\", \"errors\": {{.SensitiveTraces}}}"
  }
}

Slack Incoming Webhook example:

{
  "sensitive_trace_endpoint": {
    "name": "Slack Sensitive Traces",
    "description": "Sends sensitive traces to Slack via Incoming Webhook",
    "url": "https://hooks.slack.com/services/YOUR_WEBHOOK_URL",
    "header_template": "Content-Type: application/json",
    "body_template": "{\"attachments\": [{\"text\": {{printf \"*Prequel Sensitive Trace | Destination:* %s\\n\\n*<https://YOUR_ADMIN_HOST/destinations/%s|View Destination>*\\n\\n*Environment:* %s\\n\\n*Timestamp:* %s\\n\\n*Traces:*\\n```json\\n%s\\n```\" .DestinationId .DestinationId .Environment .Timestamp .SensitiveTraces | printf \"%q\"}}, \"color\": \"danger\", \"mrkdwn_in\": [\"text\"]}]}"
  }
}

Replace YOUR_WEBHOOK_URL with your Slack Incoming Webhook URL and YOUR_ADMIN_HOST with your Prequel admin console host (e.g. app.prequel.co).

Notes:

• The header_template is plain text with one header per line as Key: Value as shown in the example, not JSON.
• For authorization, deliveries are not signed by Prequel, unlike other Prequel webhooks. You must authenticate using your secret, as shown.
• To update or delete the endpoint at any time, use the PATCH and DELETE endpoints respectively.

Template system

The URL, headers, and body of the webhook can be customized using Go text/template syntax. The following variables are available:

Default behavior

When no custom header_template or body_template is provided, Prequel uses the following defaults:

Default Headers

Content-Type: application/json
Authorization: Bearer {Secret}

Default Body

{
  "environment": "prod",
  "sensitive_traces": [
    { "destination_id": "dest-123", "trace": "ERROR: ..." }
  ],
  "destination_id": "dest-123",
  "timestamp": "2024-01-15T10:30:00Z"
}

Sensitive trace payload schema

Each item in SensitiveTraces is an object that can include:

{
  "destination_id": "dest-123",
  "source_id": "src-456",
  "model_id": "model-789",
  "model_name": "users_table",
  "source_vendor": "postgres",
  "destination_vendor": "snowflake",
  "execution_context": "write_destination",
  "trace": "ERROR: duplicate key value violates unique constraint ..."
}

For connection tests, execution_context will reflect the test (e.g., test_source_connection or test_destination_connection). trace is always present.

Related

• Webhooks Integration
• Slack Integration
• Datadog Integration
• Logging & Monitoring