Skip to main content

Databricks setup Preview

You can configure the Databricks adapter by running dbt init in your CLI or manually providing the profiles.yml file with the fields configured for your authentication type.

The Databricks adapter for Fusion supports the following authentication methods:

  • Personal access token (for individual users)
  • Service Principal token (for service users)
  • OAuth

Databricks configuration details​

The dbt Fusion Engine dbt-databricks adapter is the only supported connection method for Databricks.

dbt-databricks can connect to Databricks SQL Warehouses. These warehouses are the recommended way to get started with Databricks.

Refer to the Databricks docs for more info on how to obtain the credentials for configuring your profile.

Configure Fusion​

Executing dbt init in your CLI will prompt for the following fields:

  • Host: Databricks instance hostname (excluding the http or https prefix)
  • HTTP Path: Path to your SQL server or cluster
  • Schema: The development/staging/deployment schema for the project
  • Catalog (Optional): The Databricks catalog containing your schemas and tables

Alternatively, you can manually create the profiles.yml file and configure the fields. See examples in authentication section for formatting. If there is an existing profiles.yml file, you are given the option to retain the existing fields or overwrite them.

Next, select your authentication method. Follow the on-screen prompts to provide the required information.

Supported authentication types​

Enter your personal access token (PAT) for the Databricks environment. For more information about obtaining a PAT, refer to the Databricks documentation. This is considered a legacy feature by Databricks and OAuth is recommended over PATs.

Example personal access token configuration​

profiles.yml
default:
  target: dev
  outputs:
    dev:
      type: databricks
      database: TRANSFORMING
      schema: JANE_SMITH
      host: YOUR.HOST.COM
      http_path: YOUR/PATH/HERE
      token: ABC123
      auth_type: databricks_cli
      threads: 16

More information​

Find Databricks-specific configuration information in the Databricks adapter reference guide.

Was this page helpful?

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

0
Edit this page