S3 - Prequel

Prerequisites

By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier:

Trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRoleWithWebIdentity"
      ],
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Condition": {
        "StringEquals": {
          "accounts.google.com:oaud": "<some_organization_identifier>",
          "accounts.google.com:sub": "<some_service_account_identifier>"
        }
      }
    }
  ]
}

Set up destination S3 bucket

Create bucket

Navigate to the S3 service page.
Click Create bucket.
Enter a Bucket name and modify any of the default settings as desired. Note: Object Ownership can be set to “ACLs disabled” and Block Public Access settings for this bucket can be set to “Block all public access” as recommended by AWS. Make note of the Bucket name and AWS Region.
Click Create bucket.

Recommendation: dedicated bucket for data transfersUse a unique bucket for these transfers. This:

Prevents resource contention with other workloads
Avoids accidental data loss from mixed lifecycle or cleanup rules
Improves security by reducing surface area and enabling tighter, destination-scoped policies

Create policy and IAM role

Create policy

Navigate to the IAM service page.
Navigate to the Policies navigation tab, and click Create policy.
Click the JSON tab, and paste the following policy, being sure to replace BUCKET_NAME with the name of the bucket chosen in Step 1.

IAM policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:DeleteObject"],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        }
    ]
}

Understanding the s3:DeleteObject requirementBy default, a connection test is performed against the destination during initial configuration and s3:DeleteObject is required to clean up test artifacts. Once the test has been performed successfully and the destination added, this action can be safely removed, as S3 destinations are append-only by default.

KMS encryption (optional)S3 destinations support buckets with KMS encryption (CMK). Encryption with SSE-C is not currently supported. For KMS encryption, add the following statement to the Statement array of your IAM policy to allow data encryption/decryption with your KMS key:

KMS policy statement

{
  "Effect": "Allow",
  "Action": [
    "kms:GenerateDataKey",
    "kms:Decrypt"
  ],
  "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
}

Replace REGION_NAME, ACCOUNT_ID, and KEY_ID with your values.

Click Next: Tags, click Next: Review.
Name the policy, add a description, and click Create policy.

IAM role (recommended)
AWS user with HMAC access key

Create role

Navigate to the IAM service page.
Navigate to the Roles navigation tab, and click Create role.
Select Custom trust policy and paste the provided trust policy to allow AssumeRole access to the new role. Click Next.
Add the permissions policy created above, and click Next.
Enter a Role name, for example, transfer-role, and click Create role.
Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the ARN value.

Role based authentication is the preferred authentication mode for S3 based on AWS recommendations, however, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred.

Navigate to the IAM service page.
Navigate to the Users navigation tab, and click Add users.
Enter a User name for the service, for example, transfer-service, click Next. Under Select AWS access type, select the Access key - Programmatic access option. Click Next: Permissions.
Click the Attach existing policies directly option, and search for the name of the policy created in the previous step. Select the policy, and click Next: Tags.
Click Next: Review and click Create user.
In the Success screen, record the Access key ID and the Secret access key.

Add your destination

Use the following details to complete the connection setup: bucket name, bucket region, and role ARN.

Permissions checklist

IAM policy on the role allows:
- s3:PutObject on arn:aws:s3:::BUCKET_NAME/*
- s3:DeleteObject on arn:aws:s3:::BUCKET_NAME/* (only required for initial connection test; may be removed after setup)
If using KMS encryption (CMK), IAM policy also allows:
- kms:GenerateDataKey and kms:Decrypt on your CMK ARN
Bucket exists in the intended region; folder prefix (if any) is configured as desired
Trust policy allows the data transfer service to assume the role

FAQ

How is the S3 connection secured?

The recommended approach is role-based access using an IAM Role with a scoped permissions policy. The role is assumed via a trust policy and short-lived credentials, so no long-lived access keys are required. Optionally, access can be configured with HMAC access keys if your policies require it. For at-rest encryption, S3-managed encryption or KMS CMKs are supported (see the KMS callout above for required actions). Grant only the minimum permissions needed (PutObject, and DeleteObject for initial connection test).

What are the `oaud` vs `sub` IDs used for?

These are identity claims used in the IAM trust policy when federating from GCP to AWS. sub uniquely identifies our Google principal in federation. oaud is an additional claim used to bind role assumption to your organization.

How is data organized in the bucket?

Data lands in Hive-style partitions per model: <folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<ext>. You can set <folder> during configuration.

What file formats are supported?

Parquet (default/recommended), CSV, and JSON/JSONL.

How are large datasets written?

Files are automatically split; multiple files may be written per model per transfer.

How do I know when a transfer completed?

Each transfer writes a manifest file per model under _manifests. The _manifests folder is created automatically at the root of the bucket. Files are written per model per transfer in the following format: _manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json.

Why do I sometimes see duplicates?

Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys prioritizing the most recent transfer window; manifest files can help bound the set of files to read.

​Prerequisites

​Create bucket

​Create policy

​Create role

​Permissions checklist

​FAQ

Prerequisites

Create bucket

Create policy

Create role

Permissions checklist

FAQ