The dbt-databricks adapter is maintained by the Databricks team. The Databricks team is committed to supporting and improving the adapter over time, so you can be sure the integrated experience will provide the best of dbt and the best of Databricks. Connecting to Databricks via dbt-spark has been deprecated.
About the dbt-databricks adapter
dbt-databricks is compatible with the following versions of dbt Core in dbt Cloud with varying degrees of functionality.
|Available starting with dbt 1.0 in dbt Cloud
|Available starting with dbt 1.1
|Available starting with dbt 1.3
The dbt-databricks adapter offers:
- Easier set up
- Better defaults: The dbt-databricks adapter is more opinionated, guiding users to an improved experience with less effort. Design choices of this adapter include defaulting to Delta format, using merge for incremental models, and running expensive queries with Photon.
- Support for Unity Catalog: Unity Catalog allows Databricks users to centrally manage all data assets, simplifying access management and improving search and query performance. Databricks users can now get three-part data hierarchies – catalog, schema, model name – which solves a longstanding friction point in data organization and governance.
To learn how to optimize performance with data platform-specific configurations in dbt Cloud, refer to Databricks-specific configuration.
To grant users or roles database permissions (access rights and privileges), refer to the example permissions page.
To set up the Databricks connection, supply the following fields:
|The hostname of the Databricks account to connect to
|The HTTP path of the Databricks cluster or SQL warehouse
|Name of Databricks Catalog (optional)