S3 Compatible

Prerequisites

• Many object storage platforms offer "S3 compatibility" enabling writing to and reading from the object storage using the S3 protocol. The S3 protocol uses an HMAC key comprised of an access ID and a secret to authenticate and write data.

Step 1: Create an HMAC Access ID and Secret

Consult your object storage platform's documentation to learn how to generate an HMAC Access ID and Secret.

Step 2: Add your destination

Use the following details to complete the connection setup: bucket host, bucket name, chosen folder name, HMAC access ID, and HMAC secret.

Permissions checklist

• HMAC Access ID and Secret have write permissions to the target bucket.
• Bucket is accessible from the service's egress IP.

FAQ

Q: How is the S3 Compatible connection secured?

A: We authenticate using the HMAC access ID and secret you provide. These credentials are used to sign requests to the S3-compatible API. Scope the credentials to only the required bucket to limit access.

Q: What file formats are supported?

A: Parquet (default/recommended), CSV, JSON, and JSONL.

Q: How are large datasets written?

A: Files are automatically split; multiple files may be written per model per transfer.

Q: How is the data organized in the bucket?

A: Data lands in Hive-style partitions per model: <folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<ext>. Multiple files may be written per model per transfer. Manifest files are written to a _manifests/ directory at the root of the configured folder path.

Q: How do I update my HMAC credentials?

A: Generate new HMAC credentials from your object storage platform and update them in the destination configuration.