dbt Cloud CI job

dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your GitHub account, GitLab account, or Azure DevOps account, you can configure continuous integration (CI) jobs to run when someone opens a new pull request in your dbt repository. For more information, refer to Configure CI for a job.

Draft pull requests do not trigger jobs. If you want jobs to run on each new commit, you need to mark your pull request as Ready for review.

GitLab Compatibility

Only GitLab users with a paid or self-hosted GitLab account can use GitLab webhooks.

Common Errors

If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to reconfigure the project to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead. Read more in Troubleshooting.

Configuring continuous integration in dbt Cloud

When you set up CI for a job, dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new pull request has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job.

Once you open a pull request, dbt Cloud builds the models affected by the code change in a temporary schema using the prefix dbt_cloud_pr_. dbt Cloud then runs the tests for these models as a check. This process provides a staging environment where you can check the run status and builds resulting from the code associated with the pull request's commit. When the CI job completes, you can see the run status directly in the pull request. The CI job enables you to deploy new code to production with confidence. The unique schema name, your project ID, and pull request ID (dbt_cloud_pr_1862_917) can be found in the run details for the given run as shown in the following image:

Viewing the temporary schema name for a run triggered by a PR

After completing the dbt run, dbt Cloud will update the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message will state whether the models and tests ran successfully or not. You can enable a setting in your git provider that makes "successful pull request checks" a requirement to merge code. And finally, once the pull request is closed or merged, dbt Cloud will delete the temporary schema from your data warehouse.

dbt Cloud might not drop the temporary schema from your data warehouse if your project has database or schema customization using the generate_database_name or generate_schema_name macros. For more information, refer to Temp PR schema limitations.

Configure CI for a job

If you want dbt Cloud to run the job whenever a pull request or commit is made, you can set up continuous integration (CI) for the job.

To set up CI:

1. Create a new job or edit an existing job to open the settings page.
2. Navigate to the Triggers section.
3. Select Continuous Integration (CI).
4. Select Run on Pull Requests? as shown in the following image.

Configuring continuous integration for a dbt Cloud Job

GitHub pull request example

The green checkmark means the dbt builds and tests were successful. The Details link shown here will navigate you to the relevant CI run in dbt Cloud.

GitHub pull request example

GitLab pull request example

The green checkmark means the dbt builds and tests were successful. Clicking the dbt Cloud pop up will navigate you to the relevant CI run in dbt Cloud.

GitLab pull request

Azure DevOps pull request example

The green checkmark means the dbt builds and tests were successful. Clicking on the dbt Cloud section navigates you to the relevant CI run in dbt Cloud.

Azure DevOps pull request

Configuring a Slim CI job

Slim CI offers an alternative to running and testing all models in your project, even when you're only changed a couple of things. Slim CI can decrease the time it takes by running and testing only modified models, which can also reduce unnecessary resource usage on your warehouse/data platform.

These components distinguish a Slim CI job from a dbt CI job:

• Must defer to a production job.
• Commands need to have a state:modified+ selector to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred job selected to compare state to. For more information, refer to Deferral and state comparision
• Must be triggered by a pull request.

Deferral and state comparison

When creating a job in dbt Cloud, you can configure your Execution Settings to defer to a previous run state by using the dropdown menu to select which production job you want to defer to.

Jobs that run on pull requests can select another job from the same project for deferral and comparison

When a job is selected, dbt Cloud will look at the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of new and modified resources.

In your job commands, you can signal to dbt to run only on these modified resources and their children by including the state:modified+ argument.

As example:

dbt build --select state:modified+

Because dbt Cloud manages deferral and state environment variables, there is no need to specify --defer or --state flags. Note: Both jobs need to be running dbt v0.18.0 or later.

To learn more about state comparison and deferral in dbt, read the docs on state.

Fresh rebuilds

As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that are fresher compared to a previous run.

More example commands in Pro-tips for workflows.

Make the necessary changes to your project and double-check if the temporary PR schemas drop after a merge or close of the pull request.

Note: dbt Cloud may not drop the temporary schema from your data warehouse if your project has database / schema customization via the generate_database_name / generate_schema_name macros. For more info, refer to Temp PR schema limitations.

Troubleshooting

If you're experiencing any issues, review some of the common questions and answers below.

Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps

If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.

First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, go to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.

Once you're in the repository page, select Edit and then Disconnect Repository at the bottom.

Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:

Select the GitHub, GitLab, or AzureDevOps tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.

Error messages that refer to schemas from previous PRs

If you receive a schema-related error message referencing a previous PR, this is usually an indicator that you are not using a production job for your deferral and are instead using self. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the Slim CI job for the current PR is kicked off.

To fix this issue, select a production job run to defer to instead of self.

Production job runs failing at the Clone Git Repository step

dbt Cloud can only checkout commits that belong to the original repository. dbt Cloud _cannot_ checkout commits that belong to a fork of that repository.

If you receive the following error message at the Clone Git Repository step of your job run:
Error message:
Cloning into '/tmp/jobs/123456/target'...
Successfully cloned repository.
Checking out to e845be54e6dc72342d5a8f814c8b3316ee220312...
Failed to checkout to specified revision.
git checkout e845be54e6dc72342d5a8f814c8b3316ee220312
fatal: reference is not a tree: e845be54e6dc72342d5a8f814c8b3316ee220312


Double-check that your PR isn't trying to merge using a commit that belongs to a fork of the repository attached to your dbt project.

CI job not triggering for Virtual Private dbt users

To trigger jobs on dbt Cloud using the API, your Git provider needs to connect to your dbt Cloud account.

If you're on a Virtual Private dbt Enterprise plan using security features like ingress PrivateLink or IP Allowlisting, registering CI hooks may not be available and can cause the job to fail silently.

Temp PR schema limitations

If your temporary pull request schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this:

You used dbt Cloud environment variables in your connection settings page

To resolve this, remove environment variables in your connections settings.

You have an empty/blank default schema

To change this, edit and fill in your default schema.

You have overridden the generate_schema_name macro

To resolve this, change your macro so that the temporary PR schema name contains the default prefix and review the guidance below:
• ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)
• ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).

You have overridden the generate_database_name macro

If you assume that the project's default connection is to a database named analytics, review the guidance below to resolve this:
• ✅ Database remains the same as the connection default (like analytics)
• ❌ Database has changed from the default connection (like dev).