A dbt Cloud production job allows you to set up a system to run a dbt job and job commands on a schedule, rather than running dbt commands manually from the command line or IDE. A job consists of commands that are "chained" together and executed as run steps. Each run step can succeed or fail, which may determine the job's run status (Success, Cancel, or Error).
Each job allows you to:
- Configure job commands
- View job run details, including timing, artifacts, and detailed run steps
- Access logs to view or help debug issues and historical invocations of dbt
- Set up notifications, and more
Job command types
Job commands are specific tasks executed by the job, and you can configure them seamlessly by either adding dbt commands or using the checkbox option in the Commands section.
During a job run, the commands are "chained" together and executed as run steps. When you add a dbt command in the Commands section, you can expect different outcomes compared to the checkbox option.
Every job invocation automatically includes the
dbt deps command, meaning you don't need to add it to the Commands list in your job settings. You will also notice every job will include a run step to reclone your repository and connect to your data platform, which can affect your job status if these run steps aren't successful.
Job outcome — During a job run, the built-in commands are "chained" together. This means if one of the run steps in the chain fails, then the next commands aren't executed, and the entire job fails with an "Error" job status.
Job outcome Generate docs on run checkbox — dbt Cloud executes the
dbt docs generate command, after the listed commands. If that particular run step in your job fails, the job can still succeed if all subsequent run steps are successful. Read Build and view your docs for more info.
Job outcome Source freshness checkbox — dbt Cloud executes the
dbt source freshness command as the first run step in your job. If that particular run step in your job fails, the job can still succeed if all subsequent run steps are successful. Read Source freshness for more info.
You can add or remove as many dbt commands as necessary for every job. However, you need to have at least one dbt command. There are few commands listed as "dbt Cloud CLI" or "dbt Core" in the dbt Command reference page page. This means they are meant for use in dbt Core or dbt Cloud CLI, and not in dbt Cloud IDE.
Use selectors as a powerful way to select and execute portions of your project in a job run. For example, to run tests for one_specific_model, use the selector:
dbt test --select one_specific_model. The job will still run if a selector doesn't match any models.
Job outcome — During a job run, the commands are "chained" together and executed as run steps. If one of the run steps in the chain fails, then the subsequent steps aren't executed, and the job will fail.
In the following example image, the first four run steps are successful. However, if the fifth run step (
dbt run --select state:modified+ --full-refresh --fail-fast) fails, then the next run steps aren't executed, and the entire job fails. The failed job returns a non-zero exit code and "Error" job status:
Job command failures
Job command failures can mean different things for different commands. Some common reasons why a job command may fail:
dbt runexecutes compiled SQL model files against the current target database. It will fail if there is an error in any of the built models. Tests on upstream resources prevent downstream resources from running and a failed test will skip them.
dbt testruns tests defined on models, sources, snapshots, and seeds. A test can pass, fail, or warn depending on its severity. Unless you set warnings as errors, only an error stops the next step.
dbt buildruns models, tests, snapshots, and seeds. This command executes resources in the DAG-specified order. If any upstream resource fails, all downstream resources are skipped, and the command exits with an error code of 1.
selectmatches multiple nodes and one of the nodes fails, then the job will have an exit code
1and the subsequent command will fail. If you specified the
—fail-fastflag, then the first failure will stop the entire connection for any models that are in progress.
If a selector doesn't match any nodes, it's not considered a failure.