> ## Documentation Index
> Fetch the complete documentation index at: https://docs.prequel.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Connect your Databricks workspace as a source

This connection uses a Databricks service principal with read-only access to the tables to import. Nothing is created, modified, or staged in your catalogs.

> [!NOTE]
>
> **Unity Catalog required**
>
> Unity Catalog is required for this integration. Tables in the legacy Hive metastore are not supported, including tables in the `hive_metastore` catalog of a Unity Catalog workspace.

## Step 1: Identify or create your SQL warehouse

Queries run on a SQL warehouse in your workspace. You can create a dedicated warehouse or use an existing one for the connection.

**1. Open the SQL warehouse**

In your Databricks workspace, click **SQL Warehouses** in the navigation pane. To create a dedicated warehouse, click **Create SQL Warehouse**, enter a **Name**, choose a **Cluster size**, and click **Create**. Otherwise, select the existing warehouse you want to use.

![Create SQL warehouse](https://storage.googleapis.com/prequel_docs/images/databricks-new-sql-warehouse-ui-refresh.png)

**2. Note the connection details**

Click the **Connection details** tab and make a note of the **Server hostname**, **Port**, and **HTTP path**. You will provide these in Step 5.

![SQL warehouse connection details](https://storage.googleapis.com/prequel_docs/images/databricks-server-path-ui-refresh.png)

## Step 2: Create a service principal

> [!NOTE]
>
> **Workspace admin required**
>
> To complete the following steps, you or a Databricks admin on your team must be a workspace admin.

**1. Add the service principal**

Click your username in the top right, click **Settings**, then **Identity and access**. Next to **Service principals**, click **Manage**. Click **Add service principal**, click **Add new**, enter a display name, and click **Add**.

**2. Grant the SQL access entitlement**

Open the new service principal. Under **Entitlements**, select **Databricks SQL access** and click **Update**. Make a note of the **Application ID**, which you will use in the following steps.

**3. Grant access to the SQL warehouse**

Return to **SQL Warehouses** and select the warehouse from Step 1. Click **Permissions**, search for and select the service principal, choose **Can use**, and click **Add**.

## Step 3: Grant read access to your data

Run the following statements in the SQL editor, replacing the catalog, schema, and table names with your own and `<application_id>` with the **Application ID** from Step 2. To run them, you must own the objects or have the `MANAGE` privilege on them. Only the tables and views you grant access to appear when you choose what to import.

```sql title="Grant read access" icon="database" expandable theme={null}
-- allow the service principal to see the catalog and schema
GRANT USE CATALOG ON CATALOG `<catalog_name>` TO `<application_id>`;
GRANT USE SCHEMA ON SCHEMA `<catalog_name>`.`<schema_name>` TO `<application_id>`;

-- grant read access to each table or view you want to import
GRANT SELECT ON TABLE `<catalog_name>`.`<schema_name>`.`<table_name_a>` TO `<application_id>`;
GRANT SELECT ON TABLE `<catalog_name>`.`<schema_name>`.`<table_name_b>` TO `<application_id>`;
```

> [!NOTE]
>
> **Granting access to the whole schema**
>
> To grant read access to every table and view in the schema instead of listing them individually, replace the per-table grants with a single grant on the schema. A schema-level grant also applies to tables and views created later.
>
> ```sql title="Grant all tables" icon="database" theme={null}
> GRANT SELECT ON SCHEMA `<catalog_name>`.`<schema_name>` TO `<application_id>`;
> ```

> [!TIP]
>
> If you provide a timestamp column for change tracking, ensure that its values increase in commit order. **Why:** A row committed later with an older timestamp can be skipped.

## Step 4: Authorize authentication

Choose the authentication method that fits your security policy. An OAuth secret is recommended because it issues short-lived tokens and can be rotated or revoked centrally.

> [tab] OAuth secret (recommended)
>
> **1. Generate a secret**
>
> In **Settings**, open **Identity and access**, click **Manage** next to **Service principals**, and open the service principal from Step 2. On the **Secrets** tab, click **Generate secret**.
>
> **2. Note the client ID and secret**
>
> Enter a lifetime for the secret, select the **all APIs** scope if prompted, click **Generate**, and make a note of the **Secret** and **Client ID**. The **Client ID** is the **Application ID** of the service principal, not the ID of the secret.
>
> > [!NOTE]
> >
> > **Select the `all APIs` scope when generating the secret**
> >
> > The OAuth connection method requires the `all-apis` scope. If Databricks prompts you to choose [scoped OAuth secrets](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m#-scoped-oauth-secrets), select **all APIs**. Secrets limited to narrower scopes such as `sql` will fail to connect.
> >
> > This scope only controls which Databricks APIs the secret can call. What the service principal can read is still governed by the warehouse permission and Unity Catalog grants from the previous steps.

> [tab] Personal access token
>
> > [!WARNING]
> >
> > A personal access token is a long-lived credential. Use this method only when your policy prevents OAuth, and store the token securely.
>
> **1. Generate your own token**
>
> As a workspace admin, generate a personal access token for yourself if you don't already have one.
>
> **2. Allow the service principal to use tokens**
>
> In **Settings**, open the **Advanced** section under **Workspace admin**. In **Access control**, next to **Personal Access Tokens**, click **Permission Settings**. Search for and select the service principal, choose **Can use**, click **Add**, and then click **Save**.
>
> **3. Generate a token for the service principal**
>
> Using your own personal access token, run the following command to create a token on behalf of the service principal. Record the `token_value` in the response.
>
> ```bash title="Create service principal token" icon="terminal" expandable theme={null}
> curl --request POST "https://<server_hostname>/api/2.0/token-management/on-behalf-of/tokens" \
>   --header "Authorization: Bearer <your_personal_access_token>" \
>   --data '{
>     "application_id": "<application_id>",
>     "lifetime_seconds": 31536000,
>     "comment": "Data import"
>   }'
> ```
>
> If PAT scopes are enabled in your workspace, add `"scopes": ["sql"]` to the request body, and use a personal access token of your own that has the `authentication` scope.

> [!NOTE]
>
> **IP access lists (optional)**
>
> If your workspace enforces [IP access lists](https://docs.databricks.com/aws/en/security/network/front-end/ip-access-list), add our static IP address to your allow list. Reach out to your account representative for the static IP address to use.

> [!NOTE]
>
> **SSH tunnel (optional)**
>
> If your workspace is not reachable over the public internet, you can connect through an SSH tunnel instead. Provide the bastion host details in Step 5, and add the public key from your account representative to the bastion's `authorized_keys` file.
>
> Only SQL queries run through the tunnel. With an OAuth secret, token requests go directly to your workspace hostname, which must remain reachable from our static IP. If your workspace is only reachable through the tunnel, use a personal access token.
>
> If your workspace also enforces IP access lists, allow the bastion's egress IP for SQL queries and our static IP for OAuth token requests.

## Step 5: Submit your connection details

Provide the following details to complete the source setup:

1. The **name** is a descriptive name of the source.
2. The **server hostname** from Step 1 (e.g., `dbc-a1b2c3d4-e5f6.cloud.databricks.com`).
3. The **port** from Step 1 (e.g., `443`).
4. The **HTTP path** from Step 1 (e.g., `/sql/1.0/warehouses/a1b2c3d4e5f6a7b8`).
5. The **catalog** from Step 3.
6. The authentication credentials from Step 4:
   * If using an **OAuth secret**: the **client ID** and **secret**.
   * If using a **personal access token**: the **token**.

If you are connecting through an SSH tunnel, also provide the **SSH host**, **SSH port**, and **SSH username** for your bastion server.

## Terraform

If your organization manages infrastructure with Terraform, you can use the configuration below instead of the steps above. It sets up the same resources and permissions, using OAuth secret authentication.

* Use the official [`databricks/databricks`](https://registry.terraform.io/providers/databricks/databricks/latest/docs) provider. Run it as a workspace admin who can grant privileges on `catalog` and `schema_name`.
* In `terraform.tfvars`, set `catalog`, `schema_name` (an existing schema), and `warehouse_size`. The optional inputs and their defaults are `table_names` (empty, which grants `SELECT` on the whole schema), `warehouse_name` (`transfer-warehouse`), `serverless` (`true`), and `service_principal_name` (`transfer-service-principal`).
* After `terraform apply`, use the `server_hostname`, `port`, `http_path`, `catalog`, `client_id`, and `client_secret` outputs to complete the connection setup. Print each with `terraform output -raw <name>`.
* The client secret expires after 730 days. The next `terraform apply` after that creates a new secret, which you then update in the connection.

```hcl title="main.tf" icon="code" expandable theme={null}
terraform {
  required_providers {
    databricks = { source = "databricks/databricks" }
  }
}

variable "catalog" {
  type        = string
  description = "Existing Unity Catalog catalog to read from."
}

variable "schema_name" {
  type        = string
  description = "Existing schema to read from."
}

variable "table_names" {
  type        = list(string)
  default     = []
  description = "Tables and views to grant SELECT on. Leave empty to grant SELECT on the whole schema, including tables and views created later."
}

variable "warehouse_name" {
  type    = string
  default = "transfer-warehouse"
}

variable "warehouse_size" {
  type        = string
  description = "Size of the new SQL warehouse, e.g. 2X-Small."
}

variable "serverless" {
  type        = bool
  default     = true
  description = "When enabled, creates a serverless SQL warehouse. Disable it in workspaces without serverless compute to create a classic warehouse."
}

variable "service_principal_name" {
  type    = string
  default = "transfer-service-principal"
}

resource "databricks_sql_endpoint" "transfer" {
  name                      = var.warehouse_name
  cluster_size              = var.warehouse_size
  auto_stop_mins            = 10
  enable_serverless_compute = var.serverless
  max_num_clusters          = 1
}

resource "databricks_service_principal" "transfer" {
  display_name          = var.service_principal_name
  databricks_sql_access = true
}

# Default scope is all-apis, which the connection requires. The secret expires
# after 730 days; the next apply after that creates a new one.
resource "databricks_service_principal_secret" "transfer" {
  service_principal_id = databricks_service_principal.transfer.id
}

# Grants Can use on the new warehouse. Terraform created this warehouse, so
# managing its whole permission set does not touch any warehouse you already have.
resource "databricks_permissions" "warehouse" {
  sql_endpoint_id = databricks_sql_endpoint.transfer.id
  access_control {
    service_principal_name = databricks_service_principal.transfer.application_id
    permission_level       = "CAN_USE"
  }
}

resource "databricks_grant" "catalog" {
  catalog    = var.catalog
  principal  = databricks_service_principal.transfer.application_id
  privileges = ["USE_CATALOG"]
}

resource "databricks_grant" "schema" {
  schema     = "${var.catalog}.${var.schema_name}"
  principal  = databricks_service_principal.transfer.application_id
  privileges = length(var.table_names) == 0 ? ["USE_SCHEMA", "SELECT"] : ["USE_SCHEMA"]
}

resource "databricks_grant" "tables" {
  for_each   = toset(var.table_names)
  table      = "${var.catalog}.${var.schema_name}.${each.value}"
  principal  = databricks_service_principal.transfer.application_id
  privileges = ["SELECT"]
}

output "server_hostname" {
  value = databricks_sql_endpoint.transfer.odbc_params[0].hostname
}

output "port" {
  value = databricks_sql_endpoint.transfer.odbc_params[0].port
}

output "http_path" {
  value = databricks_sql_endpoint.transfer.odbc_params[0].path
}

output "catalog" {
  value = var.catalog
}

output "client_id" {
  value = databricks_service_principal.transfer.application_id
}

output "client_secret" {
  value     = databricks_service_principal_secret.transfer.secret
  sensitive = true
}
```

```hcl title="terraform.tfvars" icon="code" expandable theme={null}
catalog        = "my_catalog"
schema_name    = "my_schema"
warehouse_size = "2X-Small"
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.