# Project Parsing

## Related documentation

* The `dbt parse` [command](./commands/parse.md)
* Partial parsing [profile config](../docs/local/profiles.yml.md#partial_parse) and [CLI flags](./global-configs/parsing.md)
* Parsing [CLI flags](./global-configs/parsing.md)

## What is parsing?

At the start of every dbt invocation, dbt reads all the files in your project, extracts information, and constructs a manifest containing every object (model, source, macro, etc). Among other things, dbt uses the `ref()`, `source()`, and `config()` macro calls within models to set properties, infer dependencies, and construct your project's DAG.

Parsing projects can be slow, especially as projects get bigger—hundreds of models, thousands of files—which is frustrating in development. There are a handful of ways to optimize dbt performance today:

* LibYAML bindings for PyYAML
* Partial parsing, which avoids re-parsing unchanged files between invocations
* A static parser, which extracts information from simple models much more quickly
* [RPC server](./commands/rpc.md), which keeps a manifest in memory, and re-parses the project at server startup/hangup

These optimizations can be used in combination to reduce parse time from minutes to seconds. At the same time, each has some known limitations, so they are disabled by default.

## PyYAML + LibYAML

dbt uses [PyYAML](https://pyyaml.org/wiki/PyYAML) to read and validate YAML files in your project. PyYAML is written in pure Python, but it can leverage [LibYAML](https://pyyaml.org/wiki/LibYAML) (written in C, much faster) if it's available in your system. Whenever it parses your project, dbt will always check first to see if LibYAML is available.

You can test to see if LibYAML is installed by running this command in the environment where you've installed dbt:

```text
python -c "from yaml import CLoader"
```

## Partial parsing

After parsing your project, dbt stores an internal project manifest in a file called `partial_parse.msgpack`. When partial parsing is enabled, dbt will use that internal manifest to determine which files have been changed (if any) since it last parsed the project. Then, it will *only* parse the changed files, or files related to those changes.

Starting in v1.0, partial parsing is **on** by default. In development, partial parsing can significantly reduce the time spent waiting at the start of a run, which translates to faster dev cycles and iteration.

The [`PARTIAL_PARSE` global config](./global-configs/parsing.md) can be enabled or disabled via `profiles.yml`, environment variable, or CLI flag.

(Applies to dbt v2.0 and later)

Fusion and partial parsing

Fusion job runs no longer support the `--partial-parse` and `--no-partial-parse` CLI flags. If you pass them (for example, from a dbt Core command or script), dbt logs deprecation warning `dbt1700`. Remove these flags from your Fusion job commands. For more information, refer to [Deprecated flags](../docs/dbt-versions/core-upgrade/upgrading-to-v2.md#deprecated-flags) in the guide to upgrading to the dbt Fusion engine.

### Known limitations

Parse-time attributes (dependencies, configs, and resource properties) are resolved using the parse-time context. When partial parsing is enabled, and certain context variables change, those attributes will *not* be re-resolved, and are likely to become stale.

In particular, you may see incorrect results if these attributes depend on "volatile" context variables, such as [`run_started_at`](./dbt-jinja-functions/run_started_at.md), [`invocation_id`](./dbt-jinja-functions/invocation_id.md), or [flags](./dbt-jinja-functions/flags.md). These variables are likely (or even guaranteed!) to change in each invocation. dbt Labs *strongly discourages* you from using these variables to set parse-time attributes (dependencies, configs, and resource properties).

Starting in v1.0, dbt *will* detect changes in environment variables. It will selectively re-parse only the files that depend on that [`env_var`](./dbt-jinja-functions/env_var.md) value. (If the env var is used in `profiles.yml` or `dbt_project.yml`, a full re-parse is needed.) However, dbt will *not* re-render **descriptions** that include env vars. If your descriptions include frequently changing env vars (this is highly uncommon), we recommend that you fully re-parse when generating documentation: `dbt docs generate --no-partial-parse`.

If certain inputs change between runs, dbt will trigger a full re-parse. The results will be correct, but the full re-parse may be quite slow. Today those inputs are:

* `--vars`
* `profiles.yml` content (or `env_var` values used within)
* `dbt_project.yml` content (or `env_var` values used within)
* installed packages
* dbt version
* certain widely-used macros (for example, [builtins](./dbt-jinja-functions/builtins.md), overrides, or `generate_x_name` for `database`/`schema`/`alias`)

If you're triggering [CI](../docs/deploy/continuous-integration.md) job runs, the benefits of partial parsing are not applicable to new pull requests (PR) or new branches. However, they are applied on subsequent commits to the new PR or branch.

When partial parsing is enabled, dbt may occasionally fail or incorrectly parse the project causing:

* Nodes (for example, models, sources) to not be found.
* Configurations to be set incorrectly (for example, different from what is defined in a model's `schema.yml` file).

If you get into this state, you can trigger a full re-parse using any of the following options:

* Run the dbt command with `--no-partial-parse`.
* Delete the `target/partial_parse.msgpack` file by running `dbt clean`.

You can disable partial parsing entirely by setting the `PARTIAL_PARSE` global config to `false`.

## Static parser

At parse time, dbt needs to extract the contents of `ref()`, `source()`, and `config()` from all models in the project. Traditionally, dbt has extracted those values by rendering the Jinja in every model file, which can be slow. We statically analyze model files leveraging [`tree-sitter`](https://github.com/tree-sitter/tree-sitter). You can see the code for an initial Jinja2 grammar [here](https://github.com/dbt-labs/tree-sitter-jinja2).

The static parser is **on** by default. We believe it can offer *some* speed up to 95% of projects. You may optionally turn it off using the [`STATIC_PARSER` global config](./global-configs/parsing.md).

For now, the static parser only works with models, and models whose Jinja is limited to those three special macros (`ref`, `source`, `config`). The static parser is at least 3x faster than a full Jinja render. Based on testing with data from dbt, we believe the current grammar can statically parse 60% of models in the wild. So for the average project, we'd hope to see a 40% speedup in the model parser.

## Was this page helpful?

YesNo

[Privacy policy](https://www.getdbt.com/cloud/privacy-policy)[Create a GitHub issue](https://github.com/dbt-labs/docs.getdbt.com/issues)

This site is protected by reCAPTCHA and the Google [Privacy Policy](https://policies.google.com/privacy) and [Terms of Service](https://policies.google.com/terms) apply.