BigQuery

This connection relies on two service accounts. You create a service account in your own GCP project that holds the BigQuery read permissions described below. Our service account is then granted permission to impersonate yours using short-lived tokens. As a result, no private key is ever shared, every action appears in your own audit logs, and you can revoke access at any time through your IAM settings.

The recommended authentication method, service account impersonation, is covered in Step 3. If your policy requires a different approach, that step also describes how to use a service account key.

Step 1: create a service account

Open the Service Accounts menu

In the GCP console, in the same project as your BigQuery data, navigate to the IAM & Admin menu, open the Service Accounts tab, and click Create service account.

Name the service account

Give the service account a descriptive name and click Create and Continue.

Grant the BigQuery User role

Grant the new service account the BigQuery User role. This allows it to run the read queries that extract your data.

Finish and note the email

Click Done to finish creating the account. Open the service account in the list and make a note of its email, which looks like service-account-name@project-id.iam.gserviceaccount.com. You will provide this email in Step 5.

Step 2: grant read access to the source dataset

Open the dataset sharing settings

In BigQuery, click on the dataset you want to read from. In the menu on the right, click Sharing and then click Permissions.

Add the service account as a principal

Click Add Principal.

Grant the BigQuery Data Viewer role

Add the service account you created in Step 1, grant it the BigQuery Data Viewer role, and click Save.

If you provide a timestamp column for change tracking, ensure that its values increase in commit order. Why: A row committed later with an older timestamp can be skipped.

Step 3: authorize authentication

Choose the authentication method that fits your security policy. Service account impersonation is recommended because it avoids sharing any long-lived credentials.

Service account impersonation (recommended)
Service account key

Open the service account permissions

In the IAM & Admin menu, open the Service Accounts tab and click the service account you created in Step 1 to open its details. Navigate to the Principals with access tab and click Grant Access.

Grant permission to impersonate

Add our service account as a principal and grant it both the Service Account Token Creator and Service Account User roles. Reach out to your account representative for the exact service account email to add.

Step 4: find your project ID and data location

Find the Project ID

In the Google Cloud console, open the projects list dropdown and make a note of your BigQuery Project ID.

Find the data location

Open your source dataset in BigQuery and make a note of its Data location, such as us or us-central1. You will provide this value as the region in Step 5.

This connection supports Google Cloud organization policies that restrict identities by domain. If your organization enforces domain-restricted sharing, you can add our principal to your allow list following Google’s guidance on restricting identities by domain. Contact your account representative for the customer ID to add.

Step 5: submit your connection details

Provide the following details to complete the source setup:

The name is a descriptive name of the source.
The Project ID from Step 4.
The region (the data location) from Step 4.
The authentication credentials from Step 3:
- If using impersonation: the service account email from Step 1.
- If using a service account key: the service account email from Step 1 and the JSON key you downloaded.

Terraform

If your organization manages infrastructure with Terraform, you can use the configuration below instead of the steps above. It sets up the same resources and permissions, using service account impersonation.

Use the official hashicorp/google provider. The project needs the IAM API enabled.
In terraform.tfvars, set project_id, dataset_id (an existing dataset), and vendor_service_account_email (our service account email, which your account representative provides). The optional service_account_id input defaults to transfer-source.
After terraform apply, use the project_id, region, and service_account_email outputs to complete the connection setup. Print each with terraform output -raw <name>.

variable "project_id" {
  type        = string
  description = "GCP project that holds the source dataset."
}

variable "dataset_id" {
  type        = string
  description = "Existing dataset to read from."
}

variable "vendor_service_account_email" {
  type        = string
  description = "The service account email provided for the source; it impersonates the source service account."
}

variable "service_account_id" {
  type    = string
  default = "transfer-source"
}

data "google_bigquery_dataset" "source" {
  project    = var.project_id
  dataset_id = var.dataset_id
}

resource "google_service_account" "source" {
  project      = var.project_id
  account_id   = var.service_account_id
  display_name = "Source service account"
}

resource "google_project_iam_member" "bigquery_user" {
  project = var.project_id
  role    = "roles/bigquery.user"
  member  = "serviceAccount:${google_service_account.source.email}"
}

resource "google_bigquery_dataset_iam_member" "data_viewer" {
  project    = var.project_id
  dataset_id = data.google_bigquery_dataset.source.dataset_id
  role       = "roles/bigquery.dataViewer"
  member     = "serviceAccount:${google_service_account.source.email}"
}

resource "google_service_account_iam_member" "token_creator" {
  service_account_id = google_service_account.source.name
  role               = "roles/iam.serviceAccountTokenCreator"
  member             = "serviceAccount:${var.vendor_service_account_email}"
}

resource "google_service_account_iam_member" "service_account_user" {
  service_account_id = google_service_account.source.name
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${var.vendor_service_account_email}"
}

output "project_id" {
  value = var.project_id
}

output "service_account_email" {
  value = google_service_account.source.email
}

output "region" {
  value = lower(data.google_bigquery_dataset.source.location)
}

project_id                   = "my-project"
dataset_id                   = "my_dataset"
vendor_service_account_email = "some-name@some-project.iam.gserviceaccount.com"

Getting started

Core concepts

Features

Logging, Monitoring, and Observability

Sources

Security & compliance

Developer SDKs

Step 1: create a service account

Step 2: grant read access to the source dataset

Step 3: authorize authentication

Step 4: find your project ID and data location

Step 5: submit your connection details

Terraform

​Step 1: create a service account

​Step 2: grant read access to the source dataset

​Step 3: authorize authentication

​Step 4: find your project ID and data location

​Step 5: submit your connection details

​Terraform

Step 1: create a service account

Step 2: grant read access to the source dataset

Step 3: authorize authentication

Step 4: find your project ID and data location

Step 5: submit your connection details

Terraform