> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prequel.co/llms.txt
> Use this file to discover all available pages before exploring further.

# S3

> Configuring your AWS S3 destination.

## Prerequisites

* By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with our identifier to grant access. It should look similar to the following JSON object with a proper service account identifier:

```json title="Trust policy" icon="brackets-curly" expandable theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRoleWithWebIdentity"
      ],
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Condition": {
        "StringEquals": {
          "accounts.google.com:oaud": "<some_organization_identifier>",
          "accounts.google.com:sub": "<some_service_account_identifier>"
        }
      }
    }
  ]
}
```

<Steps>
  <Step title="Set up destination S3 bucket">
    ### Create bucket

    1. Navigate to the **S3** service page.
    2. Click **Create bucket**.
    3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**.
    4. Click **Create bucket**.

    <Note>
      **Recommendation: dedicated bucket for data transfers**

      Use a unique bucket for these transfers. This:

      * Prevents resource contention with other workloads
      * Avoids accidental data loss from mixed lifecycle or cleanup rules
      * Improves security by reducing surface area and enabling tighter, destination-scoped policies
    </Note>
  </Step>

  <Step title="Create policy and IAM role">
    ### Create policy

    1. Navigate to the **IAM** service page.
    2. Navigate to the **Policies** navigation tab, and click **Create policy**.
    3. Click the **JSON** tab, and paste the following policy, being sure to replace `BUCKET_NAME` with the name of the bucket chosen in Step 1.

    ```json title="IAM policy" icon="brackets-curly" expandable theme={null}
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": ["s3:PutObject", "s3:DeleteObject"],
                "Resource": "arn:aws:s3:::BUCKET_NAME/*"
            }
        ]
    }
    ```

    <Note>
      **Understanding the s3:DeleteObject requirement**

      By default, a connection test is performed against the destination during initial configuration and `s3:DeleteObject` is required to clean up test artifacts. Once the test has been performed successfully and the destination added, this action can be safely removed, as S3 destinations are append-only by default.
    </Note>

    <Note>
      **KMS encryption (optional)**

      S3 destinations support buckets with KMS encryption (CMK). Encryption with SSE-C is not currently supported. For KMS encryption, add the following statement to the `Statement` array of your IAM policy to allow data encryption/decryption with your KMS key:

      ```json title="KMS policy statement" icon="brackets-curly" expandable theme={null}
      {
        "Effect": "Allow",
        "Action": [
          "kms:GenerateDataKey",
          "kms:Decrypt"
        ],
        "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
      }
      ```

      Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values.
    </Note>

    4. Click **Next: Tags**, click **Next: Review**.
    5. Name the policy, add a description, and click **Create policy**.

    <Tabs>
      <Tab title="IAM role (recommended)">
        ### Create role

        1. Navigate to the **IAM** service page.
        2. Navigate to the **Roles** navigation tab, and click **Create role**.
        3. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**.
        4. Add the permissions policy created above, and click **Next**.
        5. Enter a **Role name**, for example, `transfer-role`, and click **Create role**.
        6. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the **ARN** value.
      </Tab>

      <Tab title="AWS user with HMAC access key">
        Role based authentication is the preferred authentication mode for S3 based on AWS recommendations, however, HMAC Access Key ID & Secret Access Key is an alternative authentication method that can be used if preferred.

        1. Navigate to the **IAM** service page.
        2. Navigate to the **Users** navigation tab, and click **Add users**.
        3. Enter a **User name** for the service, for example, `transfer-service`, click **Next**. Under **Select AWS access type**, select the **Access key - Programmatic access** option. Click **Next: Permissions**.
        4. Click the **Attach existing policies directly** option, and search for the name of the policy created in the previous step. Select the policy, and click **Next: Tags**.
        5. Click **Next: Review** and click **Create user**.
        6. In the **Success** screen, record the **Access key ID** and the **Secret access key**.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Add your destination">
    Use the following details to complete the connection setup: **bucket name**, **bucket region**, and **role ARN**.
  </Step>
</Steps>

## Permissions checklist

* IAM policy on the role allows:
  * `s3:PutObject` on `arn:aws:s3:::BUCKET_NAME/*`
  * `s3:DeleteObject` on `arn:aws:s3:::BUCKET_NAME/*` (only required for initial connection test; may be removed after setup)
* If using KMS encryption (CMK), IAM policy also allows:
  * `kms:GenerateDataKey` and `kms:Decrypt` on your CMK ARN
* Bucket exists in the intended region; folder prefix (if any) is configured as desired
* Trust policy allows the data transfer service to assume the role

## FAQ

<AccordionGroup>
  <Accordion title="How is the S3 connection secured?">
    The recommended approach is role-based access using an IAM Role with a scoped permissions policy. The role is assumed via a trust policy and short-lived credentials, so no long-lived access keys are required. Optionally, access can be configured with HMAC access keys if your policies require it. For at-rest encryption, S3-managed encryption or KMS CMKs are supported (see the KMS callout above for required actions). Grant only the minimum permissions needed (PutObject, and DeleteObject for initial connection test).
  </Accordion>

  <Accordion title="What are the `oaud` vs `sub` IDs used for?">
    These are identity claims used in the IAM trust policy when federating from GCP to AWS. `sub` uniquely identifies our Google principal in federation. `oaud` is an additional claim used to bind role assumption to your organization.
  </Accordion>

  <Accordion title="How is data organized in the bucket?">
    Data lands in Hive-style partitions per model: `<folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<ext>`. You can set `<folder>` during configuration.
  </Accordion>

  <Accordion title="What file formats are supported?">
    `Parquet` (default/recommended), `CSV`, and `JSON`/`JSONL`.
  </Accordion>

  <Accordion title="How are large datasets written?">
    Files are automatically split; multiple files may be written per model per transfer.
  </Accordion>

  <Accordion title="How do I know when a transfer completed?">
    Each transfer writes a manifest file per model under `_manifests`. The `_manifests` folder is created automatically at the root of the bucket. Files are written per model per transfer in the following format: `_manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json`.
  </Accordion>

  <Accordion title="Why do I sometimes see duplicates?">
    Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys prioritizing the most recent transfer window; manifest files can help bound the set of files to read.
  </Accordion>
</AccordionGroup>