dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your GitHub account, GitLab account, or Azure DevOps account, you can configure jobs to run when new pull requests are opened against your dbt repo.
dbt Cloud will build the models affected by the new pull request code change in a temp schema, which acts as a quasi-staging environment, and will also run the tests that you've written for these models as a check. When the CI job completes, the run status will be shown directly in the pull request. This makes it possible to deploy new code to production with confidence.
Draft Pull Requests
Jobs will not be triggered by draft pull requests. If you would like jobs to run on each new commit, please mark your pull request as Ready for review.
GitLab Webhooks are available to only GitLab users who have a paid or self-hosted GitLab account.
If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to reconfigure the project to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead.
Understanding dbt Cloud Slim CI
When a dbt Cloud CI job is set up, dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new PR has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job. Crucially, this run will build into a temporary schema using the prefix
dbt_cloud_pr_. This schema isolation acts as a quasi-staging environment, so that you can see the builds resulting from the code associated with the PR's commit sha. The unique schema name can be found in the run details for the given run, as shown below.
When the run is complete, dbt Cloud will update the PR in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run, letting you know if the models and tests ran successfully or not. And finally, once the pull request is closed or merged, dbt Cloud will delete the temporary schema from your data warehouse.
GitHub pull request example
The green checkmark means the dbt builds and tests were successful. The Details link shown here will navigate you to the relevant CI run in dbt Cloud.
GitLab pull request example
The green checkmark means the dbt builds and tests were successful. Clicking the dbt Cloud pop up will navigate you to the relevant CI run in dbt Cloud.
Azure DevOps pull request example
The green checkmark means the dbt builds and tests were successful. Clicking pipeline navigates you to the relevant CI run in dbt Cloud.
Configuring a dbt Cloud CI job
Setting up a CI job is very similiar to setting up a normal production job that runs on a schedule; however, a CI job has some noteable differences.
There are a few components that define a Slim CI job.
- The Slim CI job must defer to a production job.
- The Slim CI job commands need to have a
state:modified+selector to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred job seleted to compare state to.
- The Slim CI job must be triggered by pull request.
Deferral and State Comparison
When creating a job in dbt Cloud, you can set your execution settings to defer to a previous run state. Use the dropdown menu to select which production job you want to defer to.
Jobs that run on pull requests can select another job from the same project for deferral and comparison
When a job is selected, dbt Cloud will look at the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of new and modified resources.
In your job commands, you can signal to dbt to run only on these modified resources and their children by including the
dbt build --select state:modified+
Because dbt Cloud manages deferral and state environment variables, there is no need to specify
--state flags. Note: Both jobs need to be running dbt v0.18.0 or later.
To learn more about state comparison and deferral in dbt, read the docs on state.
Using a webhook trigger
In the Triggers section of the jobs settings, switch to the Webhooks tab, and then check the box next to Run on Pull Requests? as shown below.
This tells dbt Cloud to run the job whenever a pull request or commit is made, rather than on a schedule. Be sure to turn the schedule of the job off if you don't want it to also run on a time-based cadence.
As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that are fresher compared to a previous run.
More example commands in Pro-tips for workflows.
Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps
If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.
First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, head to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.
Once you're in the repository page, click Edit and then click Disconnect Repository at the bottom.
Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:
Select the GitHub, GitLab, or AzureDevOps tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.
Error messages that refer to schemas from previous PRs
If you receive a schema-related error message referencing a previous PR, this is usually an indicator that you are not using a production job for your deferral and are instead using self. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the Slim CI job for the current PR is kicked off.
To fix this issue, select a production job run to defer to instead of self.