Skip to main content

Configuring Databricks PrivateLink

Limited to certain Enterprise tiers

The PrivateLink feature is available on the following dbt Cloud Enterprise tiers:

  • Business Critical
  • Virtual Private

To learn more about these tiers, contact us at sales@getdbt.com.

The following steps will walk you through the setup of a Databricks AWS PrivateLink endpoint in the dbt Cloud multi-tenant environment.

  1. Locate your Databricks Workspace ID
  2. Add the required information to the template below, and submit your request to dbt Support: 
Subject: New Multi-Tenant PrivateLink Request
- Type: Databricks
- Databricks workspace name:
- Databricks cluster AWS Region (e.g., us-east-1, eu-west-2):
- dbt Cloud multi-tenant environment (US, EMEA, AU):

dbt Labs will work on your behalf to complete the PrivateLink setup. Please allow 3-5 business days for this process to complete. Support will contact you when the endpoint is available.

  1. Once dbt Cloud support has notified you that setup is complete, register the VPC endpoint in Databricks and attach it to the workspace:
  • Register your VPC endpoint — Register the VPC endpoint using the VPC endpoint ID provided by dbt Support.
  • Create a Private Access Settings object — Create a Private Access Settings (PAS) object with your desired public access settings, and setting Private Access Level to Endpoint. Choose the registered endpoint created in the previous step.
  • Create or update your workspace — Create a workspace, or update your an existing workspace. Under Advanced configurations → Private Link choose the private access settings object created in the previous step.
Warning

If using an existing Databricks workspace, all workloads running in the workspace will need to be stopped to enable Private Link! Workloads also cannot be started for another 20 minutes after making changes. From the Databricks documentation:

After creating (or updating) a workspace, wait until it’s available for using or creating clusters. The workspace status stays at status RUNNING and the VPC change happens immediately. However, you cannot use or create clusters for another 20 minutes. If you create or use clusters before this time interval elapses, clusters do not launch successfully, fail, or could cause other unexpected behavior.

Create Connection in dbt Cloud​

Once you've completed the setup in the Databricks environment, you will be able to configure a private endpoint in dbt Cloud:

  1. Navigate to Settings → Create new project → select Databricks.
  2. You will see two radio buttons: Public and Private. Select Private.
  3. Select the private endpoint from the dropdown (this will automatically populate the hostname/account field).
  4. Configure the remaining data platform details.
  5. Test your connection and save it.
0
Edit this page