Skip to main content

Programmatic invocations

Programmatic invocations let you call dbt commands from Python scripts and applications, instead of running them in a shell. This is useful when you want to embed dbt runs into a larger application or workflow, while still using the same command surface area as the dbt Core CLI.

Common use cases include:

  • Running dbt as part of a Python application or service
  • Integrating dbt runs into orchestration workflows
  • Building internal tools that need to run dbt commands and inspect results

Refer to the dbt Core package on PyPI to install the official Python package for dbt Core if you haven't done so already.

Supported arguments​

dbtRunner.invoke accepts the same arguments as the dbt Core CLI. The first positional argument is the command (for example, run, build, test), followed by any flags and options you would normally pass on the command line.

For example, dbt.invoke(["run", "--select", "tag:my_tag"]) is equivalent to running dbt run --select tag:my_tag. There is no separate, dbtRunner‑specific list of arguments; the authoritative source for available options is the CLI help reference (dbt --help, dbt run --help, and so on) and the dbt command reference documentation.

from dbt.cli.main import dbtRunner
dbt = dbtRunner()
# equivalent ways to pass arguments
dbt.invoke(["run", "--select", "tag:my_tag"])
dbt.invoke(["run"], select="tag:my_tag")

Parallel execution not supported​

dbt-core doesn't support safe parallel execution for multiple invocations in the same process. Running multiple dbt commands concurrently in one process is unsafe and officially discouraged, and requires a wrapping process to manage subprocesses. This is because:

  • Running concurrent commands can unexpectedly interact with the data platform. For example, running dbt run and dbt build for the same models simultaneously could lead to unpredictable results.
  • Each dbt-core command interacts with global Python variables. To ensure safe operation, commands need to be executed in separate processes, for example by spawning subprocesses or using Celery for orchestration.

For safe parallel execution, you can use the dbt CLI or Studio IDE, both of which do that additional work to manage concurrency (multiple processes) on your behalf.

dbtRunnerResult​

Each command returns a dbtRunnerResult object with the following attributes:

  • success (bool): Whether the command succeeded.
  • result: When the command completes (successfully or with handled errors), it returns the command's result(s). The return type varies by command.
  • exception: When the dbt invocation encounters an unhandled error and does not complete, the exception that was raised.
  • catalog (v2 only): The catalog that the command produces when you request catalog generation.

There is a one-to-one correspondence between CLI exit codes and the dbtRunnerResult returned by a programmatic invocation:

ScenarioCLI Exit Codesuccessresultexception
Invocation completed without error0Truevaries by commandNone
Invocation completed with at least one handled error (for example, test failure or model build error)1Falsevaries by commandNone
Unhandled error. Invocation did not complete, and returns no results.2FalseNoneException

Commitments and caveats​

We're making an ongoing commitment to providing a Python entry point at functional parity with dbt Core's CLI. We reserve the right to change the underlying implementation used to achieve that goal. We expect that the current implementation will unlock real use cases in the short- and medium-term while we work on a set of stable, long-term interfaces that will ultimately replace it.

In particular, the objects returned by each command in dbtRunnerResult.result are not fully contracted, and therefore liable to change. Some of the returned objects are partially documented, because they overlap in part with the contents of dbt artifacts. As Python objects, they contain many more fields and methods than what's available in the serialized JSON artifacts. These additional fields and methods should be considered internal and liable to change in future versions of dbt-core.

Advanced usage patterns​

caution

The syntax and support for these patterns are liable to change in future versions of dbt-core.

The goal of dbtRunner is to offer parity with CLI workflows within a programmatic environment. There are a few advanced usage patterns that extend what's possible with the CLI.

Overriding parameters​

Pass in parameters as keyword arguments, instead of a list of CLI-style strings. At present, dbt will not do any validation or type coercion on your inputs. The command must be specified, in a list, as the first positional argument.

from dbt.cli.main import dbtRunner
dbt = dbtRunner()

# these are equivalent
dbt.invoke(["--fail-fast", "run", "--select", "tag:my_tag"])
dbt.invoke(["run"], select=["tag:my_tag"], fail_fast=True)

Was this page helpful?

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

0
Loading