dbt Cloud CI job
dbt Cloud makes it easy to test every single code change you make prior to deploying that new logic into production. Once you've connected your GitHub account, GitLab account, or Azure DevOps account, you can configure continuous integration (CI) jobs to run when someone opens a new pull request in your dbt repository. For more information, refer to Configure CI for a job.
Draft pull requests do not trigger jobs. If you want jobs to run on each new commit, you need to mark your pull request as Ready for review.
Only GitLab users with a paid or self-hosted GitLab account can use GitLab webhooks.
If you previously configured your dbt project by providing a generic git URL that clones using SSH, you need to reconfigure the project to connect through dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps instead. Read more in Troubleshooting.
Configuring continuous integration in dbt Cloud
When you set up CI for a job, dbt Cloud will listen for webhooks from GitHub, GitLab, or Azure DevOps indicating that a new pull request has been opened or updated with new commits. When one of these webhooks is received, dbt Cloud will enqueue a new run of the CI job.
Once you open a pull request, dbt Cloud builds the models affected by the code change in a temporary schema using the prefix
dbt_cloud_pr_. dbt Cloud then runs the tests for these models as a check. This process provides a staging environment where you can check the run status and builds resulting from the code associated with the pull request's commit. When the CI job completes, you can see the run status directly in the pull request. The CI job enables you to deploy new code to production with confidence. The unique schema name, your project ID, and pull request ID (
dbt_cloud_pr_1862_917) can be found in the run details for the given run as shown in the following image:
After completing the dbt run, dbt Cloud will update the pull request in GitHub, GitLab, or Azure DevOps with a status message indicating the results of the run. The status message will state whether the models and tests ran successfully or not. You can enable a setting in your git provider that makes "successful pull request checks" a requirement to merge code. And finally, once the pull request is closed or merged, dbt Cloud will delete the temporary schema from your data warehouse.
dbt Cloud might not drop the temporary schema from your data warehouse if your project has database or schema customization using the
generate_schema_name macros. For more information, refer to Temp PR schema limitations.
Configure CI for a job
If you want dbt Cloud to run the job whenever a pull request or commit is made, you can set up continuous integration (CI) for the job.
To set up CI:
- Create a new job or edit an existing job to open the settings page.
- Navigate to the Triggers section.
- Select Continuous Integration (CI).
- Select Run on Pull Requests? as shown in the following image.
GitHub pull request example
The green checkmark means the dbt builds and tests were successful. The Details link shown here will navigate you to the relevant CI run in dbt Cloud.GitHub pull request example
GitLab pull request example
The green checkmark means the dbt builds and tests were successful. Clicking the dbt Cloud pop up will navigate you to the relevant CI run in dbt Cloud.GitLab pull request
Azure DevOps pull request example
The green checkmark means the dbt builds and tests were successful. Clicking on the dbt Cloud section navigates you to the relevant CI run in dbt Cloud.Azure DevOps pull request
Configuring a Slim CI job
Slim CI offers an alternative to running and testing all models in your project, even when you're only changed a couple of things. Slim CI can decrease the time it takes by running and testing only modified models, which can also reduce unnecessary resource usage on your warehouse/data platform.
These components distinguish a Slim CI job from a dbt CI job:
- Must defer to a production job.
- Commands need to have a
state:modified+selector to build only new or changed models and their downstream dependents. Importantly, state comparison can only happen when there is a deferred job selected to compare state to. For more information, refer to Deferral and state comparision
- Must be triggered by a pull request.
Deferral and state comparison
When creating a job in dbt Cloud, you can configure your Execution Settings to defer to a previous run state by using the dropdown menu to select which production job you want to defer to.Jobs that run on pull requests can select another job from the same project for deferral and comparison
When a job is selected, dbt Cloud will look at the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of new and modified resources.
In your job commands, you can signal to dbt to run only on these modified resources and their children by including the
dbt build --select state:modified+
Because dbt Cloud manages deferral and state environment variables, there is no need to specify
--state flags. Note: Both jobs need to be running dbt v0.18.0 or later.
To learn more about state comparison and deferral in dbt, read the docs on state.
As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that are fresher compared to a previous run.
More example commands in Pro-tips for workflows.
Make the necessary changes to your project and double-check if the temporary PR schemas drop after a merge or close of the pull request.
Note: dbt Cloud may not drop the temporary schema from your data warehouse if your project has database / schema customization via the
generate_schema_name macros. For more info, refer to Temp PR schema limitations.
If you're experiencing any issues, review some of the common questions and answers below.
Reconnecting your dbt project to use dbt Cloud's native integration with GitHub, GitLab, or Azure DevOps
First, make sure you have the native GitHub authentication, native GitLab authentication, or native Azure DevOps authentication set up depending on which git provider you use. After you have gone through those steps, go to Account Settings, select Projects and click on the project you'd like to reconnect through native GitHub, GitLab, or Azure DevOps auth. Then click on the repository link.
Once you're in the repository page, select Edit and then Disconnect Repository at the bottom.
Confirm that you'd like to disconnect your repository. You should then see a new Configure a repository link in your old repository's place. Click through to the configuration page:
Select the GitHub, GitLab, or AzureDevOps tab and reselect your repository. That should complete the setup of the project and enable you to set up a dbt Cloud CI job.
Error messages that refer to schemas from previous PRs
To fix this issue, select a production job run to defer to instead of self.
Production job runs failing at the Clone Git Repository step
If you receive the following error message at the Clone Git Repository step of your job run:
Cloning into '/tmp/jobs/123456/target'...
Successfully cloned repository.
Checking out to e845be54e6dc72342d5a8f814c8b3316ee220312...
Failed to checkout to specified revision.
git checkout e845be54e6dc72342d5a8f814c8b3316ee220312
fatal: reference is not a tree: e845be54e6dc72342d5a8f814c8b3316ee220312
Double-check that your PR isn't trying to merge using a commit that belongs to a fork of the repository attached to your dbt project.
CI job not triggering for Virtual Private dbt users
If you're on a Virtual Private dbt Enterprise plan using security features like ingress PrivateLink or IP Allowlisting, registering CI hooks may not be available and can cause the job to fail silently.
Temp PR schema limitations
If your temporary pull request schemas aren't dropping after a merge or close of the PR, it's likely due to the below scenarios. Open and review the toggles below for recommendations on how to resolve this:
You used dbt Cloud environment variables in your connection settings page
You have an empty/blank default schema
You have overridden the
• ✅ Temporary PR schema name contains the prefix
• ❌ Temporary PR schema name doesn't contain the prefix
You have overridden the
analytics, review the guidance below to resolve this:
• ✅ Database remains the same as the connection default (like
• ❌ Database has changed from the default connection (like