Create and rotate a service principal PAT (Azure Databricks)

Use this guide to create a Databricks Personal Access Token (PAT) for a service principal on Azure Databricks, and rotate it safely to be used in a Prequel Databricks source or destination.

📘
Service Principal Types
Azure Databricks supports two types of service principals:

Azure Databricks managed service principals: Created and managed directly within Databricks. Can create PATs through the Databricks UI.

Microsoft Entra ID managed service principals: Created in Microsoft Entra ID, then imported into Databricks. Accessed via Azure API.

This guide covers creating PATs for both Microsoft Entra ID managed service principals (indicated with Steps A) and Databricks managed service principals (with Steps B).

Prerequisites

You have Azure admin access to create app registrations in Microsoft Entra ID.
You have Azure Databricks workspace admin access to manage service principals and token permissions.
You know your workspace URL (for example, https://adb-<workspace-id>.<region>.azuredatabricks.net).

Step 0: Choose your service principal type

Option A: Microsoft Entra ID managed service principal (for cross-Azure authentication)

If you need to authenticate with both Azure Databricks and other Azure resources, create a Microsoft Entra ID service principal. If you already have one with the Application (client) ID, Directory (tenant) ID, and client secret, you can skip to Step 2A.

Option B: Databricks managed service principal (Databricks-only)

If you only need Databricks access, you can create a Databricks managed service principal directly in the UI. This approach is also used in our Databricks destination setup. Skip to Step 2B for this flow.

Step 1A: Create the app registration in Microsoft Entra ID

Sign in to the Azure portal.
Navigate to Microsoft Entra ID → App registrations → New registration.

Enter a Name (e.g., "Prequel Databricks Service Principal").
Under Supported account types, select Accounts in this organizational directory only (Single tenant).
Click Register.
On the Overview page, copy and save:
- Application (client) ID
- Directory (tenant) ID

Create a client secret

In the app registration, go to Certificates & secrets → Client secrets → New client secret.
Enter a Description and select an Expires duration (e.g., 12 months).
Click Add.
Important: Copy and securely store the Value - this is your client secret and will not be shown again.

Step 2A: Add Entra ID service principal to Databricks

Import the service principal

In your Databricks workspace, click your username → Settings.
Go to Identity and access → Service principals → Add service principal.
Select Import service principal.
Paste the Application (client) ID from Step 1.
Click Add.

Grant workspace entitlements

Click on the newly created service principal.
Under Entitlements, enable:
- Workspace access
- Databricks SQL access (if needed for SQL Warehouse access)
Click Update.

Now continue to Step 3A.

Step 2B: Create Databricks managed service principal

In your Databricks workspace, click your username → Settings.
Go to Identity and access → Service principals → Manage.
Click Add service principal → Add new.
Enter a Display name and click Add.
Click on the newly created service principal and under Entitlements, enable:
- Workspace access
- Databricks SQL access (if needed for SQL Warehouse access)
Click Update and make a note of the Application ID.

Continue to Step 3B.

Step 3A: Configure token permissions for Entra ID SP

Enable PAT usage for the service principal

Configure token permissions via API using an existing admin PAT:

curl --request PATCH 'https://adb-<workspace-id>.<region>.azuredatabricks.net/api/2.0/preview/permissions/authorization/tokens' \
--header 'Authorization: Bearer <admin-personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
  "access_control_list": [
    {
      "service_principal_name": "<application-id>",
      "permission_level": "CAN_USE"
    }
  ]
}'

Replace <application-id> with your service principal's Application (client) ID from Step 1.

📘
Note: PAT workspace setting
If PAT authentication is disabled at the workspace level, users and service principals cannot create or use PATs until re-enabled. (Microsoft Learn)

Continue to Step 4A.

Step 3B: Configure token permissions for Databricks SP

In Admin Settings → Access control → Personal access tokens → Permission settings.
Search for and select your Databricks service principal.
Grant CAN USE permission.
Click Add and Save.

Continue to Step 4B.

Step 4A: Generate Microsoft Entra ID access token

Use the service principal credentials to request an Entra ID token:

# Replace values in <>. Token expires in ~1 hour.
curl -X POST "https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'client_id=<application-id>' \
  -d 'client_secret=<client-secret>' \
  -d 'grant_type=client_credentials' \
  -d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default'

Replace:

<tenant-id> with the Directory (tenant) ID from Step 1
<application-id> with the Application (client) ID from Step 1
<client-secret> with the client secret Value from Step 1

The 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default scope identifies Azure Databricks and is not workspace-specific. Do not change it. (Microsoft Learn)

📘
Token lifetime
Entra tokens are short-lived (~1 hour). Use immediately to create the PAT, or automate with CLI/SDK tools. (Microsoft Learn)

Continue to Step 5A.

Step 4B: Create PAT for Databricks SP via API

For Databricks managed service principals, you can create a PAT using the on-behalf-of tokens API with an admin user PAT.

You'll need an existing admin Personal Access Token to create a token on behalf of the service principal:

curl --request POST "https://<databricks-workspace-url>/api/2.0/token-management/on-behalf-of/tokens" \
--header "Authorization: Bearer <admin-personal-access-token>" \
--header "Content-Type: application/json" \
--data '{
  "application_id": "<service-principal-application-id>",
  "lifetime_seconds": 31536000,
  "comment": "Prequel Databricks Source (service principal)"
}'

Replace:

<databricks-workspace-url> with your workspace URL
<admin-personal-access-token> with a workspace admin's PAT
<service-principal-application-id> with the Application ID from Step 2B

📘
Alternative: UI approach
Some Databricks managed service principals may also create PATs through Admin Settings → Identity and access → Service principals → click service principal → Access tokens → Generate new token, depending on workspace configuration.

Continue to Step 6.

Step 5A: Create PAT for Entra ID SP via API

Microsoft Entra ID service principals can only create PATs through API calls, not through the Databricks UI. This is because they are API-only identities that cannot log into the workspace interface. (Microsoft Learn)

Call the Databricks Tokens API using the workspace URL and Entra access token from Step 4:

# Create a PAT for the service principal identity
curl -X POST "https://adb-<workspace-id>.<region>.azuredatabricks.net/api/2.0/token/create" \
  -H "Authorization: Bearer <entra_access_token>" \
  -H "Content-Type: application/json" \
  -d '{
        "lifetime_seconds": 31536000,
        "comment": "Prequel Databricks Source (service principal)"
      }'

The response includes a token_value that starts with dapi.... Copy and store it securely; you will not see it again. (Databricks Documentation)

📘
Alternative: Databricks CLI approach
You can also create the PAT with the Databricks CLI (databricks tokens create) after authenticating the service principal via OAuth M2M or Entra SP auth. (Microsoft Learn)

Step 6: Use the PAT in Prequel

You can now use this token value for your Prequel Databricks source or destination configuration in the Personal access token field.

Rotation procedure

Create a new PAT for the service principal (repeat Steps 4-5).
Update the Prequel source with the new PAT.
Revoke the old PAT using the UI or API:

# Using CLI: delete by token ID
databricks tokens delete <TOKEN_ID>

(Or use the Token Management API to delete a token by ID.) (Microsoft Learn, Databricks Documentation)

Troubleshooting

403 / not authorized when creating PAT

PATs may be disabled for the workspace, or the service principal/group lacks CAN USE permission. Check Admin Settings → Access control → Personal access tokens. (Microsoft Learn)

Invalid scope when requesting Entra token

Use the exact scope 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default. (Microsoft Learn)

Account-level APIs failing with PAT

PATs are for workspace-level auth. Account-level automation requires Entra/OAuth tokens. (Microsoft Learn)