Databricks

This connection uses a Databricks service principal with read-only access to the tables to import. Nothing is created, modified, or staged in your catalogs.

Unity Catalog requiredUnity Catalog is required for this integration. Tables in the legacy Hive metastore are not supported, including tables in the hive_metastore catalog of a Unity Catalog workspace.

Step 1: Identify or create your SQL warehouse

Queries run on a SQL warehouse in your workspace. You can create a dedicated warehouse or use an existing one for the connection.

Open the SQL warehouse

In your Databricks workspace, click SQL Warehouses in the navigation pane. To create a dedicated warehouse, click Create SQL Warehouse, enter a Name, choose a Cluster size, and click Create. Otherwise, select the existing warehouse you want to use.

Note the connection details

Click the Connection details tab and make a note of the Server hostname, Port, and HTTP path. You will provide these in Step 5.

Step 2: Create a service principal

Workspace admin requiredTo complete the following steps, you or a Databricks admin on your team must be a workspace admin.

Add the service principal

Click your username in the top right, click Settings, then Identity and access. Next to Service principals, click Manage. Click Add service principal, click Add new, enter a display name, and click Add.

Grant the SQL access entitlement

Open the new service principal. Under Entitlements, select Databricks SQL access and click Update. Make a note of the Application ID, which you will use in the following steps.

Grant access to the SQL warehouse

Return to SQL Warehouses and select the warehouse from Step 1. Click Permissions, search for and select the service principal, choose Can use, and click Add.

Step 3: Grant read access to your data

Run the following statements in the SQL editor, replacing the catalog, schema, and table names with your own and <application_id> with the Application ID from Step 2. To run them, you must own the objects or have the MANAGE privilege on them. Only the tables and views you grant access to appear when you choose what to import.

Grant read access

-- allow the service principal to see the catalog and schema
GRANT USE CATALOG ON CATALOG `<catalog_name>` TO `<application_id>`;
GRANT USE SCHEMA ON SCHEMA `<catalog_name>`.`<schema_name>` TO `<application_id>`;

-- grant read access to each table or view you want to import
GRANT SELECT ON TABLE `<catalog_name>`.`<schema_name>`.`<table_name_a>` TO `<application_id>`;
GRANT SELECT ON TABLE `<catalog_name>`.`<schema_name>`.`<table_name_b>` TO `<application_id>`;

Granting access to the whole schemaTo grant read access to every table and view in the schema instead of listing them individually, replace the per-table grants with a single grant on the schema. A schema-level grant also applies to tables and views created later.

Grant all tables

GRANT SELECT ON SCHEMA `<catalog_name>`.`<schema_name>` TO `<application_id>`;

If you provide a timestamp column for change tracking, ensure that its values increase in commit order. Why: A row committed later with an older timestamp can be skipped.

Step 4: Authorize authentication

Choose the authentication method that fits your security policy. An OAuth secret is recommended because it issues short-lived tokens and can be rotated or revoked centrally.

OAuth secret (recommended)
Personal access token

Generate a secret

In Settings, open Identity and access, click Manage next to Service principals, and open the service principal from Step 2. On the Secrets tab, click Generate secret.

Note the client ID and secret

Enter a lifetime for the secret, select the all APIs scope if prompted, click Generate, and make a note of the Secret and Client ID. The Client ID is the Application ID of the service principal, not the ID of the secret.

Select the all APIs scope when generating the secretThe OAuth connection method requires the all-apis scope. If Databricks prompts you to choose scoped OAuth secrets, select all APIs. Secrets limited to narrower scopes such as sql will fail to connect.This scope only controls which Databricks APIs the secret can call. What the service principal can read is still governed by the warehouse permission and Unity Catalog grants from the previous steps.

A personal access token is a long-lived credential. Use this method only when your policy prevents OAuth, and store the token securely.

Generate your own token

As a workspace admin, generate a personal access token for yourself if you don’t already have one.

Allow the service principal to use tokens

In Settings, open the Advanced section under Workspace admin. In Access control, next to Personal Access Tokens, click Permission Settings. Search for and select the service principal, choose Can use, click Add, and then click Save.

Generate a token for the service principal

Using your own personal access token, run the following command to create a token on behalf of the service principal. Record the token_value in the response.

Create service principal token

curl --request POST "https://<server_hostname>/api/2.0/token-management/on-behalf-of/tokens" \
  --header "Authorization: Bearer <your_personal_access_token>" \
  --data '{
    "application_id": "<application_id>",
    "lifetime_seconds": 31536000,
    "comment": "Data import"
  }'

If PAT scopes are enabled in your workspace, add "scopes": ["sql"] to the request body, and use a personal access token of your own that has the authentication scope.

IP access lists (optional)If your workspace enforces IP access lists, add our static IP address to your allow list. Reach out to your account representative for the static IP address to use.

SSH tunnel (optional)If your workspace is not reachable over the public internet, you can connect through an SSH tunnel instead. Provide the bastion host details in Step 5, and add the public key from your account representative to the bastion’s authorized_keys file.Only SQL queries run through the tunnel. With an OAuth secret, token requests go directly to your workspace hostname, which must remain reachable from our static IP. If your workspace is only reachable through the tunnel, use a personal access token.If your workspace also enforces IP access lists, allow the bastion’s egress IP for SQL queries and our static IP for OAuth token requests.

Step 5: Submit your connection details

Provide the following details to complete the source setup:

The name is a descriptive name of the source.
The server hostname from Step 1 (e.g., dbc-a1b2c3d4-e5f6.cloud.databricks.com).
The port from Step 1 (e.g., 443).
The HTTP path from Step 1 (e.g., /sql/1.0/warehouses/a1b2c3d4e5f6a7b8).
The catalog from Step 3.
The authentication credentials from Step 4:
- If using an OAuth secret: the client ID and secret.
- If using a personal access token: the token.

If you are connecting through an SSH tunnel, also provide the SSH host, SSH port, and SSH username for your bastion server.

Terraform

If your organization manages infrastructure with Terraform, you can use the configuration below instead of the steps above. It sets up the same resources and permissions, using OAuth secret authentication.

Use the official databricks/databricks provider. Run it as a workspace admin who can grant privileges on catalog and schema_name.
In terraform.tfvars, set catalog, schema_name (an existing schema), and warehouse_size. The optional inputs and their defaults are table_names (empty, which grants SELECT on the whole schema), warehouse_name (transfer-warehouse), serverless (true), and service_principal_name (transfer-service-principal).
After terraform apply, use the server_hostname, port, http_path, catalog, client_id, and client_secret outputs to complete the connection setup. Print each with terraform output -raw <name>.
The client secret expires after 730 days. The next terraform apply after that creates a new secret, which you then update in the connection.

terraform {
  required_providers {
    databricks = { source = "databricks/databricks" }
  }
}

variable "catalog" {
  type        = string
  description = "Existing Unity Catalog catalog to read from."
}

variable "schema_name" {
  type        = string
  description = "Existing schema to read from."
}

variable "table_names" {
  type        = list(string)
  default     = []
  description = "Tables and views to grant SELECT on. Leave empty to grant SELECT on the whole schema, including tables and views created later."
}

variable "warehouse_name" {
  type    = string
  default = "transfer-warehouse"
}

variable "warehouse_size" {
  type        = string
  description = "Size of the new SQL warehouse, e.g. 2X-Small."
}

variable "serverless" {
  type        = bool
  default     = true
  description = "When enabled, creates a serverless SQL warehouse. Disable it in workspaces without serverless compute to create a classic warehouse."
}

variable "service_principal_name" {
  type    = string
  default = "transfer-service-principal"
}

resource "databricks_sql_endpoint" "transfer" {
  name                      = var.warehouse_name
  cluster_size              = var.warehouse_size
  auto_stop_mins            = 10
  enable_serverless_compute = var.serverless
  max_num_clusters          = 1
}

resource "databricks_service_principal" "transfer" {
  display_name          = var.service_principal_name
  databricks_sql_access = true
}

# Default scope is all-apis, which the connection requires. The secret expires
# after 730 days; the next apply after that creates a new one.
resource "databricks_service_principal_secret" "transfer" {
  service_principal_id = databricks_service_principal.transfer.id
}

# Grants Can use on the new warehouse. Terraform created this warehouse, so
# managing its whole permission set does not touch any warehouse you already have.
resource "databricks_permissions" "warehouse" {
  sql_endpoint_id = databricks_sql_endpoint.transfer.id
  access_control {
    service_principal_name = databricks_service_principal.transfer.application_id
    permission_level       = "CAN_USE"
  }
}

resource "databricks_grant" "catalog" {
  catalog    = var.catalog
  principal  = databricks_service_principal.transfer.application_id
  privileges = ["USE_CATALOG"]
}

resource "databricks_grant" "schema" {
  schema     = "${var.catalog}.${var.schema_name}"
  principal  = databricks_service_principal.transfer.application_id
  privileges = length(var.table_names) == 0 ? ["USE_SCHEMA", "SELECT"] : ["USE_SCHEMA"]
}

resource "databricks_grant" "tables" {
  for_each   = toset(var.table_names)
  table      = "${var.catalog}.${var.schema_name}.${each.value}"
  principal  = databricks_service_principal.transfer.application_id
  privileges = ["SELECT"]
}

output "server_hostname" {
  value = databricks_sql_endpoint.transfer.odbc_params[0].hostname
}

output "port" {
  value = databricks_sql_endpoint.transfer.odbc_params[0].port
}

output "http_path" {
  value = databricks_sql_endpoint.transfer.odbc_params[0].path
}

output "catalog" {
  value = var.catalog
}

output "client_id" {
  value = databricks_service_principal.transfer.application_id
}

output "client_secret" {
  value     = databricks_service_principal_secret.transfer.secret
  sensitive = true
}

catalog        = "my_catalog"
schema_name    = "my_schema"
warehouse_size = "2X-Small"

Getting started

Core concepts

Features

Logging, Monitoring, and Observability

Sources

Security & compliance

Developer SDKs

Step 1: Identify or create your SQL warehouse

Step 2: Create a service principal

Step 3: Grant read access to your data

Step 4: Authorize authentication

Step 5: Submit your connection details

Terraform

​Step 1: Identify or create your SQL warehouse

​Step 2: Create a service principal

​Step 3: Grant read access to your data

​Step 4: Authorize authentication

​Step 5: Submit your connection details

​Terraform

Step 1: Identify or create your SQL warehouse

Step 2: Create a service principal

Step 3: Grant read access to your data

Step 4: Authorize authentication

Step 5: Submit your connection details

Terraform