Skip to main content

Enabling Continuous Integration (CI)

Overview

dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your GitHub account, GitLab account, or Azure DevOps account, you can configure jobs to run when new pull requests are opened against your dbt repo.

dbt Cloud will build the models affected by the new pull request code change in a temp schema, which acts as a quasi-staging environment, and will also run the tests that you've written for these models as a check. When the CI job completes, the run status will be shown directly in the pull request. This makes it possible to deploy new code to production with confidence.

Draft Pull Requests

Jobs will not be triggered by draft pull requests. If you would like jobs to run on each new commit, please mark your pull request as Ready for review.

GitLab Compatability

GitLab Webhooks are available to only GitLab users who have a paid or self-hosted GitLab account.

Common Errors

If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to reconfigure the project to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead.

Understanding dbt Cloud Slim CI

When a dbt Cloud CI job is set up, dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new PR has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job. Crucially, this run will build into a temporary schema using the prefix dbt_cloud_pr_. This schema isolation acts as a quasi-staging environment, so that you can see the builds resulting from the code associated with the PR's commit sha. The unique schema name can be found in the run details for the given run, as shown below.

Viewing the temporary schema name for a run triggered by a PR

Viewing the temporary schema name for a run triggered by a PR

When the run is complete, dbt Cloud will update the PR in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run, letting you know if the models and tests ran successfully or not. And finally, once the pull request is closed or merged, dbt Cloud will delete the temporary schema from your data warehouse.

GitHub pull request example

The green checkmark means the dbt builds and tests were successful. The Details link shown here will navigate you to the relevant CI run in dbt Cloud.

GitHub pull request example

GitHub pull request example

GitLab pull request example

The green checkmark means the dbt builds and tests were successful. Clicking the dbt Cloud pop up will navigate you to the relevant CI run in dbt Cloud.

GitLab pull request

GitLab pull request

Azure DevOps pull request example

The green checkmark means the dbt builds and tests were successful. Clicking pipeline navigates you to the relevant CI run in dbt Cloud.

Azure DevOps pull request

Azure DevOps pull request

Configuring a dbt Cloud CI job

Setting up a CI job is very similiar to setting up a normal production job that runs on a schedule; however, a CI job has some noteable differences.

There are a few components that define a Slim CI job.

  • The Slim CI job must defer to a production job.
  • The Slim CI job commands need to have a state:modified+ selector to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred job seleted to compare state to.
  • The Slim CI job must be triggered by pull request.

Deferral and State Comparison

When creating a job in dbt Cloud, you can set your execution settings to defer to a previous run state. Use the dropdown menu to select which production job you want to defer to.

Jobs that run
on pull requests can select another job from the same project for deferral and comparison

Jobs that run on pull requests can select another job from the same project for deferral and comparison

When a job is selected, dbt Cloud will look at the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of new and modified resources.

In your job commands, you can signal to dbt to run only on these modified resources and their children by including the state:modified+ argument.

As example:

dbt build --select state:modified+

Because dbt Cloud manages deferral and state environment variables, there is no need to specify --defer or --state flags. Note: Both jobs need to be running dbt v0.18.0 or later.

To learn more about state comparison and deferral in dbt, read the docs on state.

Using a webhook trigger

In the Triggers section of the jobs settings, switch to the Webhooks tab, and then check the box next to Run on Pull Requests? as shown below.

This tells dbt Cloud to run the job whenever a pull request or commit is made, rather than on a schedule. Be sure to turn the schedule of the job off if you don't want it to also run on a time-based cadence.

Fresh Rebuilds

As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that are fresher compared to a previous run.

More example commands in Pro-tips for workflows.

Troubleshooting

Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps

If your dbt project relies the generic git clone method that clones using SSH and deploy keys to connect to your dbt repo, you need to disconnect your repo and reconnect it using the native GitHub, GitLab, or Azure DevOps integration in order to enable dbt Cloud Slim CI.

First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, head to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.

Once you're in the repository page, click Edit and then click Disconnect Repository at the bottom.

Disconnect repo

Disconnect repo

Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:

Configure repo

Configure repo

Select the GitHub, GitLab, or AzureDevOps tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.

Error messages that refer to schemas from previous PRs

If you receive a schema-related error message referencing a previous PR, this is usually an indicator that you are not using a production job for your deferral and are instead using self. If the prior PR has already been merged, the prior PR's schema may have been dropped by the time the Slim CI job for the current PR is kicked off.

To fix this issue, select a production job run to defer to instead of self.