Amazon S3

By default, S3 authentication uses role-based access. You create an IAM role in your own AWS account that holds the read permissions described below, and our identity is granted permission to assume it. No long-lived credentials are shared, and you can revoke access at any time through your IAM settings.

Files do not need to contain a timestamp column for change tracking. When the data to sync is configured, the object’s last modified time, as recorded by the storage provider, will be available as a column.

Step 1: configure AWS authentication

Choose the authentication method that fits your security policy.

IAM role (recommended)
AWS access keys

Create an IAM policy

In the AWS IAM console, navigate to the Policies tab and click Create policy. Click the JSON tab and paste the following policy, replacing BUCKET_NAME with the name of the bucket you want to read from.

IAM policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKET_NAME"
        }
    ]
}

If your S3 bucket uses KMS encryption (CMK), add the following statement to the Statement array to allow decryption with your KMS key. Replace REGION_NAME, ACCOUNT_ID, and KEY_ID with your values.

KMS decrypt statement

{
  "Effect": "Allow",
  "Action": "kms:Decrypt",
  "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
}

Name the policy, add a description, and click Create policy.

Create an IAM role

In the AWS IAM console, navigate to the Roles tab and click Create role. Select Custom trust policy and paste the trust policy below to allow our identity to assume the role. Reach out to your account representative for the value of <some_service_account_identifier>.

Trust policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRoleWithWebIdentity"
      ],
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": "<some_service_account_identifier>"
        }
      }
    }
  ]
}

Click Next, attach the permissions policy you created in the previous step, enter a Role name, for example, source-s3, and click Create role.

Note the role ARN

Once the role is created, search for it in the Roles list, click the role name, and make a note of its ARN (for example, arn:aws:iam::123456789012:role/source-s3). You will provide this ARN in Step 2.

Create an IAM user

In the AWS IAM console, navigate to the Users tab and click Add users. Enter a User name for the service, for example, source-service, and select programmatic access.

Attach the policy

Attach the read policy described in the IAM role tab (s3:GetObject and s3:ListBucket on the target bucket) directly to the user.

Generate access keys

Generate an access key pair for the user and make a note of the access key ID and the secret access key. You will provide these values in Step 2.

If your bucket policy restricts access by IP, allow the static IP so we can reach the bucket. Reach out to your account representative for the static IP address to use.

Step 2: submit your connection details

Provide the following details to complete the source setup:

The name is a descriptive name of the source.
The bucket name of the bucket you want to read from.
The AWS region the bucket is hosted in.
The authentication credentials from Step 1:
- If using an IAM role: the role ARN.
- If using AWS access keys: the access key ID and the secret access key.

Terraform

If your organization manages infrastructure with Terraform, you can use the configuration below instead of the steps above. It sets up the same resources and permissions, using IAM role authentication.

Use the official hashicorp/aws provider, version 6.0 or later.
In terraform.tfvars, set bucket_name (an existing bucket) and service_account_identifier (the accounts.google.com:sub value, which your account representative provides). The optional inputs and their defaults are role_name (source-s3), policy_name (source-s3-read), and kms_key_arn (empty).
After terraform apply, use the bucket_name, bucket_region, and role_arn outputs to complete the connection setup. Print each with terraform output -raw <name>.

variable "bucket_name" {
  type        = string
  description = "Name of the existing bucket to read from."
}

variable "service_account_identifier" {
  type        = string
  description = "The service account identifier (accounts.google.com:sub) provided for the trust policy."
}

variable "role_name" {
  type    = string
  default = "source-s3"
}

variable "policy_name" {
  type    = string
  default = "source-s3-read"
}

variable "kms_key_arn" {
  type        = string
  default     = ""
  description = "ARN of the KMS key (CMK) encrypting the bucket, if any."
}

data "aws_s3_bucket" "source" {
  bucket = var.bucket_name
}

data "aws_iam_policy_document" "trust" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = ["accounts.google.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "accounts.google.com:sub"
      values   = [var.service_account_identifier]
    }
  }
}

data "aws_iam_policy_document" "read" {
  statement {
    effect    = "Allow"
    actions   = ["s3:GetObject"]
    resources = ["${data.aws_s3_bucket.source.arn}/*"]
  }
  statement {
    effect    = "Allow"
    actions   = ["s3:ListBucket"]
    resources = [data.aws_s3_bucket.source.arn]
  }
  dynamic "statement" {
    for_each = var.kms_key_arn == "" ? [] : [var.kms_key_arn]
    content {
      effect    = "Allow"
      actions   = ["kms:Decrypt"]
      resources = [statement.value]
    }
  }
}

resource "aws_iam_policy" "read" {
  name   = var.policy_name
  policy = data.aws_iam_policy_document.read.json
}

resource "aws_iam_role" "source" {
  name               = var.role_name
  assume_role_policy = data.aws_iam_policy_document.trust.json
}

resource "aws_iam_role_policy_attachment" "source" {
  role       = aws_iam_role.source.name
  policy_arn = aws_iam_policy.read.arn
}

output "bucket_name" {
  value = data.aws_s3_bucket.source.bucket
}

output "bucket_region" {
  value = data.aws_s3_bucket.source.bucket_region
}

output "role_arn" {
  value = aws_iam_role.source.arn
}

bucket_name                = "my-source-bucket"
service_account_identifier = "<some_service_account_identifier>"

Getting started

Core concepts

Features

Logging, Monitoring, and Observability

Sources

Security & compliance

Developer SDKs

Step 1: configure AWS authentication

Step 2: submit your connection details

Terraform

​Step 1: configure AWS authentication

​Step 2: submit your connection details

​Terraform

Step 1: configure AWS authentication

Step 2: submit your connection details

Terraform