full_refresh

The full_refresh config allows you to control whether a resource will always or never perform a full-refresh. This config overrides the --full-refresh command-line flag.

Models

dbt_project.yml

models:
  <resource-path>:
    +full_refresh: false | true 

models/<modelname>.sql

{{ config(
    full_refresh = false | true
) }}

select ...

Seeds

dbt_project.yml

seeds:
  <resource-path>:
    +full_refresh: false | true

Description

The full_refresh config allows you to optionally configure whether a resource will always or never perform a full-refresh. This config is an override for the --full-refresh command line flag used when running dbt commands.

You can set the full_refresh config in the dbt_project.yml file or in a resource config.

full_refresh value	Behavior
If set to true	The resource always performs a full refresh, regardless of whether you pass the --full-refresh flag in the dbt command.
If set to false	The resource never performs a full refresh, regardless of whether you pass the --full-refresh flag in the dbt command.
If set to none or omitted	The resource follows the behavior of the --full-refresh flag. If the flag is used, the resource will perform a full refresh; otherwise, it will not.

Loading table...

Note

• The --full-refresh flag also supports a short name, -f.
• The should_full_refresh() macro has logic encoded.

Usage

Incremental models

• How do I rebuild an incremental model?
• What if the columns of my incremental model change?

Seeds

The columns of my seed changed, and now I get an error when running the `seed` command, what should I do?

If you changed the columns of your seed, you may get a Database Error:

Snowflake

$ dbt seed
Running with dbt=1.6.0-rc2
Found 0 models, 0 tests, 0 snapshots, 0 analyses, 130 macros, 0 operations, 1 seed file, 0 sources

12:12:27 | Concurrency: 8 threads (target='dev_snowflake')
12:12:27 |
12:12:27 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN]
12:12:30 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 2.78s]
12:12:31 |
12:12:31 | Finished running 1 seed in 10.05s.

Completed with 1 error and 0 warnings:

Database Error in seed country_codes (seeds/country_codes.csv)
  000904 (42000): SQL compilation error: error line 1 at position 62
  invalid identifier 'COUNTRY_NAME'

Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Redshift

$ dbt seed
Running with dbt=1.6.0-rc2
Found 0 models, 0 tests, 0 snapshots, 0 analyses, 149 macros, 0 operations, 1 seed file, 0 sources

12:14:46 | Concurrency: 1 threads (target='dev_redshift')
12:14:46 |
12:14:46 | 1 of 1 START seed file dbt_claire.country_codes...................... [RUN]
12:14:46 | 1 of 1 ERROR loading seed file dbt_claire.country_codes.............. [ERROR in 0.23s]
12:14:46 |
12:14:46 | Finished running 1 seed in 1.75s.

Completed with 1 error and 0 warnings:

Database Error in seed country_codes (seeds/country_codes.csv)
  column "country_name" of relation "country_codes" does not exist

Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

In this case, you should rerun the command with a --full-refresh flag, like so:

dbt seed --full-refresh

Why is this the case?

When you typically run dbt seed, dbt truncates the existing table and reinserts the data. This pattern avoids a drop cascade command, which may cause downstream objects (that your BI users might be querying!) to get dropped.

However, when column names are changed, or new columns are added, these statements will fail as the table structure has changed.

The --full-refresh flag will force dbt to drop cascade the existing table before rebuilding it.

Recommendation

• Set full_refresh: false for models of especially large datasets, which you would never want dbt to fully drop and recreate.

• You cannot override an existing full_refresh config. To change its behavior in certain circumstances, remove the config logic or update it using variables so the behavior can be overridden when needed. For example, if you have an incremental model with the following config:

  {{ config(
      materialized = 'incremental',
      full_refresh = var("force_full_refresh", false)
  ) }}

  Then override the full_refresh config to true using the --vars flag: dbt run --vars '{"force_full_refresh": true}'.

Reference docs

• on_configuration_change

Was this page helpful?

Privacy policyCreate a GitHub issue

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.