Skip to main content

dbt Cloud CI job

Overview

dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your GitHub account, GitLab account, or Azure DevOps account, you can configure jobs to run when new pull requests are opened against your dbt repo.

dbt Cloud will build the models affected by the new pull request code change in a temp schema, which acts as a quasi-staging environment, and will also run the tests that you've written for these models as a check. When the continuous integration (CI) job completes, the run status will be shown directly in the pull request. This makes it possible to deploy new code to production with confidence.

Draft Pull Requests

Jobs will not be triggered by draft pull requests. If you would like jobs to run on each new commit, please mark your pull request as Ready for review.

GitLab Compatibility

GitLab Webhooks are available to only GitLab users who have a paid or self-hosted GitLab account.

Common Errors

If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to reconfigure the project to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead.

Understanding dbt Cloud Slim CI

When a dbt Cloud CI job is set up, dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new PR has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job. Crucially, this run will build into a temporary schema using the prefix dbt_cloud_pr_. This schema isolation acts as a quasi-staging environment, so that you can see the builds resulting from the code associated with the PR's commit sha. The unique schema name can be found in the run details for the given run, as shown below.

Viewing the temporary schema name for a run triggered by a PR

Viewing the temporary schema name for a run triggered by a PR

After completing the dbt run, dbt Cloud will update the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message will state whether the models and tests ran successfully or not. You can enable a setting in your git provider that makes "successful pull request checks" a requirement to merge code. And finally, once the pull request is closed or merged, dbt Cloud will delete the temporary schema from your data warehouse.

GitHub pull request example

The green checkmark means the dbt builds and tests were successful. The Details link shown here will navigate you to the relevant CI run in dbt Cloud.

GitHub pull request example

GitHub pull request example

GitLab pull request example

The green checkmark means the dbt builds and tests were successful. Clicking the dbt Cloud pop up will navigate you to the relevant CI run in dbt Cloud.

GitLab pull request

GitLab pull request

Azure DevOps pull request example

The green checkmark means the dbt builds and tests were successful. Clicking on the dbt Cloud section navigates you to the relevant CI run in dbt Cloud.

Azure DevOps pull request

Azure DevOps pull request

Configuring a dbt Cloud CI job

Setting up a CI job is very similiar to setting up a normal production job that runs on a schedule; however, a CI job has some noteable differences.

There are a few components that define a Slim CI job.

  • The Slim CI job must defer to a production job.
  • The Slim CI job commands need to have a state:modified+ selector to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred job selected to compare state to.
  • The Slim CI job must be triggered by pull request.

Deferral and State Comparison

When creating a job in dbt Cloud, you can set your execution settings to defer to a previous run state. Use the dropdown menu to select which production job you want to defer to.

Jobs that run
on pull requests can select another job from the same project for deferral and comparison

Jobs that run on pull requests can select another job from the same project for deferral and comparison

When a job is selected, dbt Cloud will look at the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of new and modified resources.

In your job commands, you can signal to dbt to run only on these modified resources and their children by including the state:modified+ argument.

As example:

dbt build --select state:modified+

Because dbt Cloud manages deferral and state environment variables, there is no need to specify --defer or --state flags. Note: Both jobs need to be running dbt v0.18.0 or later.

To learn more about state comparison and deferral in dbt, read the docs on state.

Using a webhook trigger

In the Triggers section of the jobs settings, switch to the Webhooks tab, and then check the box next to Run on Pull Requests? as shown below.

Configuring webhooks for a dbt Cloud Job

Configuring webhooks for a dbt Cloud Job

This tells dbt Cloud to run the job whenever a pull request or commit is made, rather than on a schedule. Be sure to turn the schedule of the job off if you don't want it to also run on a time-based cadence.

Fresh Rebuilds

As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that are fresher compared to a previous run.

More example commands in Pro-tips for workflows.

Limitations

If your temporary PR schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this:

You used dbt Cloud environment variables in your connection settings page
To resolve this, remove environment variables in your connections settings.
You have an empty/blank default schema
To change this, edit and fill in your default schema.
You have overridden the generate_schema_name macro
To resolve this, change your macro so that the temporary PR schema name contains the default prefix and review the guidance below:
• ✅ Temporary PR schema name contains the prefix dbt_cloud_pr_ (like dbt_cloud_pr_123_456_marketing)
• ❌ Temporary PR schema name doesn't contain the prefix dbt_cloud_pr_ (like marketing).
You have overridden the generate_database_name macro
If you assume that the project's default connection is to a database named analytics, review the guidance below to resolve this:
• ✅ Database remains the same as the connection default (like analytics)
• ❌ Database has changed from the default connection (like dev).

Make the necessary changes to your project and double-check if the temporary PR schemas drop after a merge or close of the PR.

Troubleshooting

Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps

If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.

First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, head to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.

Once you're in the repository page, click Edit and then click Disconnect Repository at the bottom.

Disconnect repo

Disconnect repo

Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:

Configure repo

Configure repo

Select the GitHub, GitLab, or AzureDevOps tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.

Error messages that refer to schemas from previous PRs

If you receive a schema-related error message referencing a previous PR, this is usually an indicator that you are not using a production job for your deferral and are instead using self. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the Slim CI job for the current PR is kicked off.

To fix this issue, select a production job run to defer to instead of self.

Production job runs failing at the Clone Git Repository step

dbt Cloud can only checkout commits that belong to the original repository. dbt Cloud cannot checkout commits that belong to a fork of that repository.

If you receive the following error message at the Clone Git Repository step of your job run:

Error message:
Cloning into '/tmp/jobs/123456/target'...
Successfully cloned repository.
Checking out to e845be54e6dc72342d5a8f814c8b3316ee220312...
Failed to checkout to specified revision.
git checkout e845be54e6dc72342d5a8f814c8b3316ee220312
fatal: reference is not a tree: e845be54e6dc72342d5a8f814c8b3316ee220312

Double-check that your PR isn't trying to merge using a commit that belongs to a fork of the repository attached to your dbt project.

CI job not triggering for Virtual Private dbt users

To trigger jobs on dbt Cloud using the API, your Git provider needs to connect to your dbt Cloud account.

If you're on a Virtual Private dbt Enterprise plan using security features like ingress PrivateLink or IP Allowlisting, registering CI hooks may not be available and can cause the job to fail silently.

0